System and method for collecting network performance information

ABSTRACT

A network communications device including an input/output (I/O) unit configured to communicate data packets to and from a communications network and at least one module. The module(s) may be configured to count total data packets being communicated via the I/O unit during a time period, count data packets including real-time content being communicated via the I/O unit during the time period, determine bandwidth of total data packets being communicated via the I/O unit, determine bandwidth of data packets including real-time content being communicated via the I/O unit, and communicate indicia representative of the bandwidth of data packets including real-time content and total data packets via said I/O unit to another network communications device. The module(s) may be software, hardware, or a combination thereof.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a Continuation-in-Part of U.S. patent applicationSer. No. 11/479,751 filed on Jun. 30, 2006, Ser. No. 11/583,765 filed onOct. 19, 2006, and Ser. No. 11/583,288 filed on Oct. 19, 2006; thisapplication also claims priority to Provisional Patent Applications60/839,333 filed on Aug. 22, 2006, 60/897,543 filed on Jan. 26, 2007,and 60/905,624 filed on Mar. 7, 2007, and 60/922,246 filed on Apr. 5,2007; the entire teachings of which are incorporated herein byreference.

BACKGROUND OF THE INVENTION

Telephony has advanced dramatically with the advancement of technology.Telephone communication was once limited to an analog public switchedtelephone network (PSTN), where the PSTN has been traditionally formedof two types of telephone carriers, local and long distance. The localcarriers established local networks for subscribers to communicatewithin local regions, and the long distance carriers created networksbetween the local networks to enable subscribers of different localcarriers to communicate with one another.

Over time, mobile telephone networks were developed to enablesubscribers to use mobile telephones. At first, the wireless networksand handsets were analog. Technology for the wireless networks wasdeveloped to provide digital wireless communications, which provided aclearer signal than analog wireless communications.

About the same time that the digital wireless networks were developing,the Internet was also developing. The International StandardsOrganization (ISO) developed an Open Systems Interconnection (OSI) basicreference model in 1977 that currently includes seven different layers.Each of the layers provides protocols for certain types of operations.More specifically, the seven layers include: physical layer (Layer 1),data link layer (Layer 2), network layer (Layer 3), transport layer(Layer 4), session layer (Layer 5), presentation layer (Layer 6), andapplication layer (Layer 7). Each entity interacts directly with thelayer immediately beneath it and provides facilities for use by thelayer above it. The protocols on each layer enable entities tocommunicate with other entities on the same layer. The Internetinitially provided for simple digital data to be communicated betweenusers. One of the early communication application included email.However, as communications standards and protocols developed, theInternet matured to include more advanced communication applications,including voice over Internet protocol (VoIP).

FIG. 1 is an illustration of a legacy telecommunications network thatincludes class 4 and 5 switches 102 a-102 n (collectively 102) and 104a-104 n (collectively 104), respectively, connected to a signalingsystem #7 (SS7) network 106 (indicated as dashed lines). Historically,the class 5 switches 104 were generally configured to communicate viain-band signaling verses the use of SS7 signaling and operate to form alocal network of subscribers within the network to place telephone callsto one another. The class 4 switches 102 were developed for longdistance connections between the class 5 switches 104 at end offices(not shown). The class 4 switches 102, which are monolithic, aregenerally formed of multiple components, including a port, portcross-connect matrix, switch messaging bus with external signalingunits, and call processing unit, as understood in the art. Class 4switches are circuit based and utilize time division multiplexing (TDM)and are capable of terminating higher high-speed communications,including T1, T3, OC-3, and other four-wire circuit connections. Asunderstood in the art, TDM is a synchronous communications protocol.

The SS7 network 106, which includes signal transfer points (STPs) 108a-108 n (collectively 108), service switching points (SSs) on the class4 and class 5 switches, and service control points (SCPs—not shown). TheSS7 network is connected to the class 4 and 5 switches for providing andmaintaining inter-switch call services between the switches. The SS7network is used to signal out-of-band call setup, as out-of-bandsignaling is more secure and faster than in-band signaling. The callstate changes of the inter-switch trunks of the class 4 switch arecommunicated to the adjoining switches via the SS7 network via aconnection to the STP. To manage and route calls, the STPs 108 are usedas an Inter-Switch messaging network whereby two switches control thetrunking between the switches via messaging over the SS7 networkprovided by the STP switches that act as the inter-switch message bus. Acall state machine of the class 4 switch provides control for routingtraffic within the cross-connect matrix of the monolithic switch. Thecall state machine also provides call control signaling information toother switches via the connections to the STPs. The call controlsignaling information is routed via the STPs to other switches for callsetup and tear-down. The call control information routed by the STPscontains pertinent information about the call to allow the terminatingswitch to complete various calls.

Telephony has benefited from the development of the OSI model in a vastnumber of ways. One way has been through separating the call controllerinto a distributed cross-connect on an asynchronous network, such asasynchronous transfer mode (ATM) or a Internet Protocol (IP) network.FIG. 2 is an illustration of a conventional telephony network 200 thatincludes a packet network 202. In one embodiment, the packet network 202is an ATM network. Media gateways (MGs) 204 a-204 n (collectively 204)are media translation or conversion devices that modify and convertprotocols between disparate communication networks. The media gateways204, which are in communication with class 5 switches 206 a-206 n(collectively 206) are located at the edge of the packet network 202.The media gateways 204 convert TDM packets or streams 208 into packets,frames, or cells (collectively referred to hereinafter as “packets”) 210and vice versa.

The packet network 202 operates independently as a distributed virtualmedia gateway port cross-connect for voice calls primarily due to one ormore call control managers (CCMs) 212 located on the packet network 202.The call control manager 212 is in communication with the media gateways204 and operates to control the media gateways 204 and provideinstructions on how to rotate the packets 210 via far-end addressallocations. By separating the call controller from the class 4switches, the packet network 202 becomes, in effect, the virtualcross-connect of the switching system. The packet network 202 enablespackets 210 that include voice data, commonly known as bearer packets,to be tagged with a destination address 214 a and origination address214 b for enabling content data 214 c to be properly routed from theorigination media gateway 204 b to the destination media gateway 204 a.The media gateways 204 use of the packet network 202 is controlled bythe CCM 212 and may communicate the packets 210 over the packet network202 via IP addresses and virtual circuit (VC) or virtual path (VP)between the media gateways 204 to appropriately route the packets to thecorrect destination network node through the packet network 202. CCM 212receives call state processing information from the media gateways 204and signaling points, and processes the call state changes by using lookup tables (not shown). The CCM 212 thereafter communicates packetaddressing and state changes to the media gateways 204 to process thecall.

Ethernet protocol was developed to provide for a computer network thatenables multiple computers to share a common externalinter-communication bus. Ethernet is generally used to provide for localarea networks (LANs). Ethernet operates by communicating frames of data.While Ethernet operates well within a local environment (e.g., within abuilding) because Ethernet assumes that there is an known capacity ofbandwidth associated with the bus standards set forth in the IEEE 802.3standard that defines Ethernet. Ethernet is a shared environment, whereco-utilization creates transmission errors called collisions. Thesecollisions are detected by Ethernet cards in computers and a randomre-transmission timer is used to avoid the next collision. Ethernetposes special problems for use in communications systems given it lacksdedicated bandwidth and time slots. The shared nature of an Ethernetnetwork creates additional complexities in that the amount of availablebandwidth can vary when used with wireless technologies.

Communication protocols transmitted over packet networks, such as ATM orIP networks, may utilize TDM based transmission facilities, which aresynchronous as compared to Ethernet transmission facilities, which areasynchronous. Synchronous transmission protocols utilize a common clockand channel schema so that each device on the network operatessynchronously with a dedicated path. Two types of “connection” stateknowledge are present in a dedicated system, such as a TDM. Each channelhas a dedicated amount of bandwidth and an error rate that is calculatedfrom a common clock to determine path errors. The two types ofconnection state awareness functionality are provided by the channelitself and the common clock and data within a TDM header. The commonclock provides for a determination of (i) a communications data ratefrom one end-point to another end-point and (ii) the data quality.Additionally, the TDM protocol includes “far end state” data in a headerof a TDM frame to indicate whether there is a connection at the far end,thereby providing an indication of continuity along the communicationspath. Specifically, in-band end-to-end alarming allows the cross-connectdevices to receive indications of continuity problems with otherend-points. The in-band alarming is also provided for connectionquality, where Bit Error Rate (BER) allows each end-point to know thequality of the data being received. Furthermore, bandwidth is always inuse meaning that packets are synchronous, which that the far end knowsexactly how many packets are to be sent and received in a given timeperiod (e.g., one second). Computation of utilization is easily made byusing the known bandwidth and multiplying it by the “seizure” time orthe amount of time in use.

Packet-based communications sessions lack circuit based connection stateawareness indicators and clocking functionality to provide a sessioncontroller the ability to know the path connectivity state toefficiently manage making call handling decisions with anything otherthan ample bandwidth to setup and use sessions. This lack of connectionpath state awareness with the communications protocols, such as Ethernetand Internet protocol (IP) technologies, result in “gaps” in terms ofbeing able to react to decaying transmission path quality and besensitive to shared use of bandwidth. Most IP call controller solutionsare founded on enterprise applications, where a single entity owns thenetwork and scale of the network is relatively small. IP and Ethernetprotocols lack the in-band path signaling, quality and use metrics toallow for this scale, or the ability to perform enhanced call handlingwith paths outside the governance of the packet network. Because packetcommunications are asynchronous, there is no common clock, and, thus,there is no way to know how many packets were transmitted, which, inturn, removes the ability to characterize transmission quality of theentire path, the amount of bandwidth available, or the amount in use.Further, packet networks are “converged” meaning they have bothreal-time and non-real-time bandwidth use. Currently, there is noin-band mechanism for determining real-time and non-real-time bandwidthuse; having such information would allow for handling calls. It iscommonly understood that proper connection operational assumptions aremade by call control engines when the SS7 signaling path is properlyoperating (e.g., provisioned bandwidth is available) between end-pointswithin the SS7 network. These operational assumptions are problematic asEthernet, IP, and other data networks become oversubscribed and causethe packet network to become congested and prevent throughput. In caseswhere an end-point, such as a WiFi telephone, is mobile and bandwidthchanges with signal strength (e.g., a WiFi telephone losing bandwidth asan individual walks away from a connection point antenna), theconnection operation assumptions also fail to provide graceful callhandling.

One available technique in packet networks to prevent oversubscriptionof real-time media traffic is through the use of call admission control(CAC) or the IP equivalent known as Resource Reservation Protocol(RSVP). CAC is primarily used to prevent congestion in voice traffic andis applied in the call setup phase to ensure there is enough bandwidthfor data flow by reserving resources. To reserve bandwidth through theentire packet network a CAC requires that the CAC procedure be performedat each point along a virtual circuit between two media gateways onwhich a call is to be routed, and often in a bi-directional fashion.While CAC functionality exists, the use of such CAC functionality isalmost never applied because of the amount of time needed by the CACprocedure during call set up. For example, currently, CAC typicallycannot operate over 40 calls per second and typical call set-ups onmedia gateways or class 4 switches may be 200 calls per second orhigher.

One technique used to monitor the performance of IP session performance(i.e., after a call session has been established) is the use of thereal-time control protocol (RTCP) as defined in IETF RFC 3550. RTCPcollects statistics on a media connection, including bytes sent, packetssent, lost packets, jitter, feedback, and round trip delay. Otherinformation may be provided in the RTCP packet using profile specificextensions. RTCP, which operates on a per session basis, is used forquality of service (QoS) reporting after termination of a session. Thestatistics information may be used, for example, to improve the qualityof service by limiting data flow or changing CODEC compression.Utilization of the real-time QoS statistics, however, is limited to thespecific session associated with the RTCP stream.

An emerging standard that is being developed for Ethernet performancemeasures is 802.1AG. This standard operates by generating andcommunicating an 802.1AG packet or “heart beat” over an Ethernet networksegment. The 802.1AG packets are communicated via a Layer 2 EthernetVirtual Circuit, such as a VLAN or Ethernet tunnel. At the ends andmid-points in Ethernet tunnels, 802.1AG packets are transmittedperiodically over the Ethernet network to the far end. The Y.1731protocol is utilized to calculate the number of data frames communicatedbetween the 802.1AG packets. This configuration enables a performancemeasures (PM) to compute certain information about the performance ofthe path between the end-points on an Ethernet network. This combinationof 802.1AG and Y.1731 enables the end points to be knowledgeable aboutthe Frame Loss Rate (FLR), packet delay, and jitter in the path. Thisconfiguration is helpful to assist in monitoring performance of anEthernet network path and diagnosing connectivity faults. However, theconfiguration falls short of providing the amount of real-time bandwidthin use or the total bandwidth in use. This information is useful to theproper management by a session controller handling calls during periodsof flux in the packet transmission path, or the management of thereal-time traffic.

Service providers often have trouble isolating and diagnosing networkproblems. To attempt to locate a packet loss problem along a nodesegment (i.e., a path between two network communications devices) over anetwork, a probe that may be used to trace data packets beingcommunicated over the node segment. This probe, however, is typically anexternal device from the network communications devices and operates torun a trace over an instant of time to determine network performanceinformation, such as packet loss, jitter, and delay. An operator usingthe external probe may view results of a trace to diagnose the networkcommunications problem. These results are not accessible to the networkcommunications devices and cannot be accessed by network communicationsdevices to alter network communications.

Telecommunications switching systems today provide for Internet protocol(IP) communications between two end-points within a network or adifferent network to be terminated to a far end-point. Calls between twoend-points are routed to the terminating end-point based on the addressinput at the originator. This address information is then relayed to aCall Control Manager (CCM) that screens, translates, and routes the callto the terminating subscriber or to another network to be terminated ata far end subscriber's end-point. The basic functionality of thisprocess is widely known within the art and is used throughouttelecommunications networks for voice calling.

Within the architecture of this switching system, calls to and fromend-points are controlled by the CCM. The CCM may be located within amonolithic device in a TDM switch architecture or provided by anoutboard computing device that controls the calls by using signalingthat controls network based routing and switching devices located withinthe network. The latter device is known as soft-switch architecture.

The soft-switch architecture within an IP network controls callprocessing through use of signaling to and from the end devices andmedia gateways. One example of a protocol used for this IP signaling isSession Initiation Protocol (SIP). This protocol is currently usedmainly with IP telephony, such as VoIP, and can be used as an accessprotocol between the end-user and the CCM and/or between the CCM of onenetwork and the CCM of another network.

Another protocol used mainly between the CCM and a media gateway is theITU-T H.248 protocol, commonly known as Megaco. This protocol is acontrol protocol that allows the CCM to control ingress and egressfrom/to the media gateway as calls are set up using a media gateway.Within a packet network framework, IP communications between twoend-points (both access end-points and media gateways) are controlled bythe signaling of the end-point to/from the CCM. The CCM providesauthentication, screening, translations and routing based on informationthat is stored in the CCM and from the state of the end-points that theCCM controls.

Within the soft-switch architecture, call control can only beaccomplished based on information possessed by the CCM or the on/offstate of the devices that has an association with the CCM. While thisconfiguration is fine in a static environment, packet networks are in astate of change at all times since the network itself can carrydifferent types of information besides voice calls. One skilled in theart knows that a packet network is a converged network that can carryvoice, data, and video all in a single path, and routing of calls withina packet network is not static and can vary significantly from call tocall.

Because of packet network content communications variables, calls mayencounter congestion and loss of voice quality based on latency, jitter,and packet loss. These content communications variables can affect anyportion of a call at any time based on the network elements usage at thetime of the call. Unlike a TDM system where dedicated channels andcircuits are provided, the CCM only has control of it own end-points.Other end units may attempt calls, computers may send/receive datawithout talking to the CCM, and other devices may require bandwidthwhile the original call is progressing, thus causing voice qualityproblems for the participants. In addition to these basic gaps, manyphysical layer 1 systems that are poor in regulating bandwidth, arebeing used for transmission facilities. WiFi, EVDO, 4G (WiMax), DSL, andcable systems are all physical layer 1 technologies that demonstratedifferent bandwidth rates and management of their ability to modifyavailable bandwidth as the Signal-to-Noise (SNR) ratios change.

Conventional soft-switching has not been designed to provide relief forcallers when congestion, jitter or delay problems, such as thosedescribed above, are encountered. Since conventional CCMs can onlydetermine call success based on connectivity to and from the callingparties, voice quality between two parties is not taken intoconsideration for call success.

Communication problems of in-band signals over packet networks aredifficult to isolate. Currently, if a communication problem exists overa transmission path, there are few techniques to isolate the problem.One technique includes using an external probe to capture and decodepackets, commonly know as a trace, traversing over a communications pathto help isolate the problem. However, technicians generally only run thetrace in response to a customer notifying a communications carrier of acommunications problem. If a problem exists across packet networksoperated by different carriers, a typical response by a carrier is tocontact the other to determine if the other carrier can locate a problemin its network. In other words, locating an in-band communicationsproblem over one or more packet networks is difficult as troubleshootingtools for such problems are limited to out-of-band performance metrics(PM) and are not available as in-band information via control orsignaling paths.

A problem that exists with current implementations of telephony overpacket networks is that a call control manager does not have informationabout the bearer path. Traditionally, there was a linkage betweentransmission path state and the monolithic switch that essentially ownedone end of that path where the in-band signaling and linecharacteristics were available and was an integral part of theinformation used by the CCM for call processing. As demonstrated incurrent implementations of VoIP, without knowledge of the bearer path,the call control manager may establish calls that result in poor voicequality or call setup failure.

In addition, IP Service gateways, such as a broadband remote accessserver (BRAS), functions to limit, commonly known as traffic rateshaping, each customer's DSL traffic to their purchased speed. There isno end-to-end signaling, outside of the embedded TCP flow controlmechanism, used to adjust the bursting to eliminate packet loss. Rateshaping is a statically forced bandwidth constraint that alters thenature of a transmission path in the packet networks. This shapingcoupled with commonly shared or “over-subscribed” bandwidth normallyassociated with trunking facilities between networks results in unknowntransmission path states between media gateways servicing VoIP and otherreal-time services, such as Video on Demand (VOD).

Traffic Quality of Service (QoS) management of packets is performed,where multiple flows aggregate into a smaller flows or channels. Theapplication of Internet Protocol QoS is performed at the egress pointwhere traffic is transmitted over a single link. Current traffic enginesuse the following information to make QOS traffic decisions. Thedecisions are assigning a Class of Service (CoS) and then acting uponthat service to shape, restrict, or pass traffic to an egress point. Thevariables used to assign priority to traffic flows can be based on:entrance port (assign a whole port a CoS), virtual circuit in a port(assign a CoS to an Ethernet Virtual Circuit, LSP, etc.), priority bitmarking of each packet (P bit), protocol type (assigning a CoS tospecific types of packets or traffic), IP address and port (assigning aCoS to a whole IP address, or its port addresses), sessionidentification (a HTTP, UDP, or other session addressed call), orotherwise. This priority marking information is used by service points,and shared links to implement QoS for the shared traffic flows. QoS andCoS types of information are made available at the point of aggregationwhere traffic management or QoS functions occur. However, the number ofpackets transmitted or lost in the packet stream elsewhere in thenetwork is currently unavailable without the use of a session or pathbased protocol. These packet loss functions are generally not tracked byQoS mechanisms.

In current traffic rate shaping designs, the Internet may burst a packetstream to a DSL user when the packet network or Digital Subscriber LineAccess Multiplexer (DSLAM) itself may not have sufficient bandwidth toaccommodate the packet session. In a TCP-based session, the transmissionrate is throttled down after packet loss is detected in the session. InVoIP, the packet loss is not counted by the use of Real Time ControlProtocol (RTCP) signaling, but it is captured as the call progresses bythe end points. RTCP, however, only considers performance of its ownsessions and not the transmission path performance as a whole. In bothcases, packets are sent over the packet network that get dropped inmid-path and will not make it to the customer premises equipment (CPE)and user. More importantly, there is no cross-session information aboutthe packet loss and no whole path information available in-band.

Also, packet loss can be due to available bandwidth transmission ratefall-off, such as when a WiFi user walks away from a WiFi Access Point(AP) and loses RF signal strength, signal-to-noise ratio increases, orcongestion increases due to many users concurrently accessing the AP.All of these types of conditions in the transmission paths can havesevere impacts upon the ability to accomplish call processing and callmanagement.

An Internet Service Provider (ISP) may provide different Internetconnectivity speeds or data transfer rates based upon their serviceplans. For example, a user may purchase 1.5 Mbits/sec data transfer ratefor a predetermined amount, such as 10 Mbits/sec data transfer rate orhigher. In general, the transmission path is between the shared(trunked) BRAS resource and the DSLAM that is supplying. The normalamount of bandwidth consumption in the download direction from thenetwork to the user is high as compared to the upstream direction.However, there is no correlated throttling mechanism in the IPweb-server linked to user's ISP service plan that can be used to shapethe packet transmissions. So, all of the network traffic is shaped atthe BRAS typically based on the user's purchased data transfer rates.Depending upon network conditions, some of this traffic may not make itto the DSL user since the BRAS has no knowledge of the IP service pathfrom itself to the customer.

A problem occurs when the BRAS does realize congestion on a packetnetwork, where packets are being dropped due to insufficient bandwidth.Some packets could be dropped in the packet network or at an aggregationdevice somewhere in the packet network. Currently, there is littleintelligence that recognizes the dropped packets in the packet network.In fact, packet networks are designed to discard traffic based on QoSmarkings. This problem is made worse because transporting packets thatwill ultimately be dropped adds to congesting the network. The packetsconsume bandwidth until dropped.

Transmission Control Protocol (TCP) was designed to work in abest-effort, packet store-and-forward environment characterized by thepossibility of packet loss, packet disordering, and packet duplication.Packet loss can occur, for example, by a congested network elementdiscarding a packet. Here, a microprocessor or memory of a networkelement may not have adequate capacity to address all packets routinginto and out of the element. Packet disordering can occur, for example,by routing changes occurring during a transmission. Here, packets of TCPconnection may be being arbitrarily transmitted partially over a lowbandwidth terrestrial path and as routing table changes occur partiallyover a high bandwidth satellite path. Packet duplication can occur, forexample, when two directly-connected network elements use a reliablelink protocol and the link goes down after the receiver correctlyreceives a packet but before the transmitter receives an acknowledgementfor the packet.

An embedded capability within TCP protocol is the TCP sliding windowtechnique. The sliding window was developed and deployed as a flowcontrol mechanism used to minimize the inefficiencies ofpacket-by-packet transmission. The sending of data between TCP enabledend-devices on a connection is accomplished using the sliding windowtechnique. TCP requires that all transmitted data be acknowledged by thereceiving host. The sliding windows method is a process by whichmultiple packets of data can be affirmed with a single acknowledgement.

SUMMARY OF THE INVENTION

To overcome the problem of limited packet network path performanceinformation, the principles of the present invention provide forcommunicating performance information packet (PIP) data packetsperiodically between network nodes to enable the network nodes tocollect and utilize network performance information. Note that PIP, asdefined in this document, is not the same as IETF RFC 1622 PIP. In oneembodiment, the PIP data packets may be 802.1AG data packets and amodified ITU-T Y.1731 protocol may be used to collect the networkperformance information. Performance information packets may betransmitted in real-time or near-real-time and received at a far-endnetwork device for the purpose of creating transmission performanceinformation. This network performance information may be made availableas a path and/or node operational measure to other applications,controllers, servers, and protocol stacks. In another embodiment, theinformation communicated in the PIP packets received at one end of thepath may be periodically sent to the other end so that the remote endhas knowledge of transmit and receive performance information. Thenetwork performance information may include total real-time bandwidthuse, total bandwidth use, packets communicated, packet loss, jitter,delay of the data packets (or other if defined) being communicatedbetween network nodes, or other information. In addition, the principlesof the present invention provide for collecting historical data rateperformance to enable management of data being communicated to thenetwork nodes based on historical data rates. For example, real-timedata rates over the course of a day may be used to provide load sharingbetween different network nodes prior to establishing communicationroutes. The principles of the present invention further provide forcollecting network performance information or statistics of data packettypes by differentiating between data packets having real-time andnon-real-time content. By collecting statistics of the different datapacket types, additional network management may be performed to avoidoverloading network nodes and degrading or dropping data packets at thenetwork nodes. In one embodiment, a call control manager may communicatewith user devices, access nodes, or service points, such as mediagateways, to gain line and trunk state “packet transmission path”information to be used for call session management by incorporating thecollected real-time data packet statistics.

One embodiment of a system in accordance with the principles of thepresent invention includes a network communications device including aninput/output (I/O) unit configured to communicate data packets to andfrom a communications network and at least one module. The module(s) maybe configured to count total data packets being communicated via the I/Ounit during a time period, count data packets including real-timecontent being communicated via the I/O unit during the time period,determine bandwidth of total data packets being communicated via the I/Ounit, determine bandwidth of data packets including real-time contentbeing communicated via the I/O unit, and communicate indiciarepresentative of the bandwidth of data packets including real-timecontent and total data packets via said I/O unit to another networkcommunications device. The module(s) may be software, hardware, or acombination thereof.

A method in accordance with the principles of the present invention mayinclude monitoring transmission characteristics on a packet network.Data packets including real-time content being communicated over anetwork node segment of a packet network may be monitored. At least onetransmission performance characteristic of data packets includingreal-time content communicated over the network node segment of thepacket network may be determined. Indicia representative of thetransmission performance characteristic(s) of the communicated datapackets including real-time content may be communicated to a networknode located on the packet network for monitoring and managing thetransmission performance characteristic(s) of the network node segment.In one embodiment, a secondary measure of burstiness or “peakedness” maybe included to indicate the ability to oversubscribe that real-timecontent flow. Peakedness refers to the standard deviation of the trafficburst from the average rate of the traffic flowing through the path. Inessence, multiple real-time sessions of different bandwidth sizes andpacketization rates can introduce bursty real-time traffic that can becharacterized and captured as path state information. Managing peakutilization enables network management to provide efficientcommunications on a packet network.

A network communications device may include an input/output unitconfigured to communicate data packets over a network node segment and amemory unit configured to store network performance informationassociated with data packets including real-time content communicatedvia the I/O unit. The stored information may include a trigger to exportthe information to the far-ends. Given that packet flows may not besymmetrical, a path state should be known at both ends of acommunications path. In one embodiment, the path state may be sharedwith network nodes along a communications path using performanceinformation packets, which may occur in a real-time or near real-timemanner.

BRIEF DESCRIPTION OF THE DRAWINGS

Illustrative embodiments of the present invention are described indetail below with reference to the attached drawing figures, which areincorporated by reference herein and wherein:

FIG. 1 is an illustration of a legacy telecommunications network thatincludes class 4 and 5 switches connected to a signaling system #7 (SS7)network;

FIG. 2 is an illustration of a conventional telephony network thatincludes a packet network;

FIG. 3 is an illustration of an exemplary packet network that utilizesperformance information packets to determine network performanceinformation;

FIG. 4 is an illustration of an exemplary data packet stream includingPIP data packets and data packets including real-time and non-real-timecontent;

FIG. 5 is a block diagram of an exemplary network node configured toperform functionality in accordance with the principles of the presentinvention;

FIG. 6 is a block diagram of exemplary modules configured to determineand collect network performance information in accordance with theprinciples of the present invention;

FIG. 7 is an illustration of multiple exemplary data packet networkshaving exemplary network nodes configured to determine and collectnetwork performance information;

FIG. 8 is a block diagram of an exemplary end or mid-point deviceshowing structural and functional operations used for employing theprinciples of the present invention;

FIG. 9 is a block diagram of an exemplary pin-hole firewall deviceshowing structural and functional operations used for employing theprinciples of the present invention;

FIG. 10 is a block diagram of an exemplary head-end device showingstructural and functional operations used for employing the principlesof the present invention;

FIG. 11 is a block diagram of exemplary modules configured to determinenetwork performance information associated with data packetscommunicated with network communication devices described in FIGS. 8-10;

FIG. 12 is an illustration of exemplary processes performed on networknodes in a data packet network;

FIG. 13 is an illustration of an exemplary network node configured toperform functionality and communications over a packet network inaccordance with the principles of the present invention;

FIG. 14 is a flow chart of an exemplary process for managing networkcommunications;

FIG. 15 is an illustration of an exemplary packet network having a callcontrol manager with a centralized table of network performanceinformation for use in managing call communications over the packetnetwork;

FIG. 16 is a flow chart of an exemplary process for using networkperformance information stored in a centralized table for controllingcalls by a call control manager;

FIG. 17A is a flow chart of a high-level process for controllingcommunications on a packet network;

FIG. 17B is one embodiment of a permission table that may be utilized toestablish permission or access levels by various network participants tonetwork performance information that has been collected over one or morenetworks;

FIG. 18 is a block diagram of exemplary multi-node packet networks usedto communicate data packets including network performance informationgenerated by each node in a transmission path;

FIG. 19 is a flow diagram of an exemplary process for generating andcommunicating network performance information in data packets inaccordance with the principles of the present invention;

FIG. 20 is a flow diagram of an exemplary process for isolating a nodewithin a packet network that generated network performance informationindicating a transmission performance problem;

FIG. 21 is an exemplary process for identifying communication problemswithin one or more packet networks;

FIG. 22 is an illustration of an exemplary packet network with oneservice provider and two operators;

FIGS. 23A and 23B are illustrations of a multi-carrier network havingmultiple Ethernet service providers (ESPs) and a multi-point networkhaving a multi-point device in communication with network interfacedevices;

FIGS. 24A-24C (collectively FIG. 24) are flow charts of an exemplaryprocess for performing line-to-line call flow;

FIGS. 25A-25C (collectively FIG. 25) are flow diagrams of an exemplaryprocess for providing call processing for rerouting a call between anoriginating line and terminating trunk;

FIGS. 26A-26C (collectively FIG. 26) is a flow chart of an exemplaryprocess for performing congestion control for calls coming through an IPtrunk to a line;

FIG. 27A is an illustration of an exemplary network system that includestwo networks operated by different communications carriers;

FIG. 27B is an illustration of an exemplary billing entity for use indetermining billing for customers and partners of a communicationscarrier;

FIGS. 28A and 28B (collectively FIG. 28) are screenshots of exemplaryweb browser interfaces;

FIG. 29 is an illustration of an exemplary graphical user interface(GUI) that displays a schematic of a packet network and performancemonitoring devices;

FIG. 30 is a screenshot of another exemplary graphical user interfacethat is displaying a chart of node segments status usage for aparticular node on a network;

FIG. 31 is an illustration of the OSI 7-layer basic reference model;

FIG. 32 is an illustration of an example of an Operations,Administration, and Maintenance Entities depicting multipleadministrative domains;

FIG. 33 illustrates a block diagram of a network entity according to anembodiment of the present invention;

FIG. 34 illustrates a plurality of Vector Performance Tables accordingto an embodiment of the present invention;

FIG. 35 illustrates a flow diagram of the MEF Maintenance Entity dataflow according to an embodiment of the present invention;

FIGS. 36-39 illustrate exemplary MEF Maintenance Entity payload ingressand egress data flows according to an embodiment of the presentinvention;

FIG. 40 illustrates an end station ME payload data flow according to anembodiment of the invention;

FIG. 41 illustrates a network diagram of a Vector PerformanceCorrelation Engine (VPCE) according to an embodiment of the presentinvention;

FIGS. 42 a-42 c illustrate a Graphical User Interface (GUI) according toan embodiment of the present invention;

FIG. 43 illustrates a MEF network implementation according to anembodiment of the present invention;

FIG. 44 illustrates a MEF network implementation of inter-layercommunication between Data Link Layer devices and Physical Layer devicesaccording to an embodiment of the present invention;

FIG. 45 illustrates a wireline digital subscriber loop network accordingto an embodiment of the present invention;

FIG. 46 illustrates a bit swapping table according to an embodiment ofthe present invention;

FIG. 47 illustrates a wireless network according to an embodiment of thepresent invention;

FIG. 48 illustrates a MEF network implementation of inter-layercommunication between Data Link Layer devices and Network Layer devicesaccording to another embodiment of the present invention;

FIG. 49 illustrates a MEF network implementation of inter-layercommunication between Data Link Layer devices and Transport Layerdevices according to another embodiment of the present invention;

FIG. 50 illustrates a TCP packet according to an embodiment of thepresent invention;

FIG. 51 illustrates a MEF network implementation of inter-layercommunication between Data Link Layer devices and Session Layer devicesaccording to another embodiment of the present invention;

FIG. 52 illustrates a MEF network implementation of inter-layercommunication between Data Link Layer devices and Presentation Layerdevices according to another embodiment of the present invention;

FIG. 53 illustrates a MEF network implementation of inter-layercommunication between Data Link Layer devices and Application Layerdevices according to another embodiment of the present invention;

FIG. 54 illustrates a block flow diagram for a TCP Window sizing methodaccording to an embodiment of the present invention;

FIG. 55 illustrates a block flow diagram for a TCP Window sizing methodaccording to another embodiment of the present invention;

FIG. 56 illustrates a network diagram including for traffic shaping anetwork including a BRAS and a DSLAM according to an embodiment of thepresent invention;

FIG. 57 illustrates a user interface for network traffic shaping methodaccording to an embodiment of the present invention;

FIG. 58 illustrates and embodiment of a Data Link Layer device and anASIC device that is associated with an incoming network interface forcommunicating to an outgoing network interface;

FIG. 59 illustrates a block flow diagram for the traffic shaping methodaccording to an embodiment of the present invention;

FIG. 60 illustrates a block flow diagram for a method of for usinginformation contained in a PIP packet to control packet traffic flowwith UDP;

FIG. 61 is an example of an Ethernet network in accordance with anillustrative embodiment of the present invention;

FIG. 62 is an example of an Ethernet network in accordance with anillustrative embodiment of the present invention;

FIG. 63 is an example of a CAC engine configuration in accordance withan illustrative embodiment of the present invention;

FIG. 64 is an example of PIP packet flow of network performanceinformation in accordance with an illustrative embodiment of the presentinvention;

FIG. 65 is an example of stored network performance informationassociated with access nodes in accordance with an illustrativeembodiment of the present invention;

FIG. 66 is a flowchart of a process for allocating network resources inaccordance with an illustrative embodiment of the present invention; and

FIG. 67 is a flowchart of a process for correcting failure of networkresources in accordance with an illustrative embodiment of the presentinvention.

DETAILED DESCRIPTION OF THE DRAWINGS

FIG. 3 is an illustration of an exemplary packet network 300 thatutilizes performance information packets 302 a, 302 b (collectively 302)communicated in-band, and along virtual packet paths between representedas node links 303 a, 303 b, and 303 c (collectively 303) between networknodes 304 a, 304 b, and 304 c (collectively 304). For purposes of thisapplication, a performance information packet (a “PIP packet” or “PIPdata packet”) shall mean a packet communicated over data paths of a datapacket network that is used by the data packet network to obtainperformance information associated with path transmission states of thedata packet network. In one embodiment, such PIP packets arecommunicated in-band along the data or bearer path of a packet network.However, such PIP packet information may also be communicatedout-of-band between network elements of the packet network to provideutilization performance measures to other switching and control systemsvia control signaling over an Operational Support Network or otheroperations or maintenance network.

A PIP packet may be communicated between the nodes of a network toestablish windows of time in which a node collects or determines networkperformance information, which may be any information that describespacket utilization and performance of a node, node segment, transmissionpath, or network element. More particularly, a PIP packet may have atimestamp, counter, sequence number or other identifiers to enable theuse of the PIP packet by a network node to establish a sampling windowof time for collecting or determining such network performanceinformation. Alternatively, a PIP packet may not include such identifierand may instead be generated at regular intervals between nodes of thenetwork. Each network node or path transmission point may transmit PIPpackets to a far-end element via the packet transmission path and thefar-end element may receive, calculate performance and store theinformation for use as a utilization and performance measurement ornetwork performance information. Given each communication path maycontain information from its transmission to receive paths, theend-points may exchange and track the measures of the bi-directionalpath via relaying those measures at either given intervals or any othermechanism so that at least one end of the communication path has boththe transmit and receive path utilization and performance measurements.The PIP packet may provide a “heartbeat” between network nodes by whichthe network nodes may use to determine the network performance. A PIPpacket may also be used to communicate the collected or determinednetwork performance information between nodes of the network by, forexample, including the network performance information in the header orpayload portion of the PIP packet. In one embodiment, a PIP packet is anEthernet Connectivity Fault Management (CFM) packet, such as an 802.1AGpacket, and a receiving utilization and performance tracking mechanismmay be a ITU Y.1731 protocol stack. However, any packet of data may beutilized under any suitable protocol, as well as calculation methodologyto track and store the network performance information.

The PIP packets 302 may be formatted to include any information capableof providing information to network nodes to determine networkperformance information, where the network performance information mayinclude transmission rate, transmission quality, and/or transmissionconnectivity. Other network performance information, such ascommunication path utilization, may additionally be determined.

The PIP data packets 302 provide a “heartbeat” to enable the networknodes 304 at the far-end (i.e., receiving end) to generate networkperformance information. The PIP data packets 302 may be communicatedevery T_(PIP) seconds, where T_(PIP) may be less than, equal to, orhigher than one second and include a timestamp of the time ofcommunication and a counter value indicating the number of packetspreviously sent to enable a receiving network node to determine whetherany data packets were lost between successive PIP data packets.Transmission rate is an indication of the number of data packetscommunicated over a time period and can be determined by counting datapackets communicated over a network segment during a time period, forexample. The PIP data packets may be used in determining a time periodover which to measure the transmission rate. Transmission quality is ameasure of link state and may include various link state parameters,including packet loss, jitter, and delay, for example. In oneembodiment, the PIP data packets 302 may be communicated over Layer 2 ofthe OSI model and the network performance information may be determinedin Layer 2. The network nodes 304 may include transmission performancecollection units 306 a, 306 b, and 306 c (collectively 306),respectively, to generate and collect network performance information.Transmission connectivity is an indication of communications between twodevices on a network. The connectivity may be indicative of signalstrength of communications between the devices, an on/off indication ofa device being off or otherwise incapable of communicating, or othersuitable performance measures. The transmission performance units 306may generate network performance information in association with the PIPdata packets 302.

Network performance information may also include information associatedwith non-packet networks, such as cable DOCSIS, wireless CDMA, TDMA,WiMax, or circuit based networks. Without limiting the foregoing,network performance information may include any data that is associatedwith any wired or wireless network that may be indicative of orotherwise suitable for use in determining the operation or generalhealth of the network. Network performance information may also includeinformation that is not associated with the communication of databetween network elements along connection paths, but is insteadassociated with the performance of network devices themselves located ata particular node of the network. For example, network performanceinformation may indicate buffer utilization levels, buffer overflows,errors experienced in caching or queuing data, latency introduced by alack of processing, packet loss across a switching fabric of aparticular network device such as a switch and router, or any otherperformance issue associated with a particular network device. It shouldbe understood that different network types and different connectionpaths may have different indicia of network performance issues. Forexample, a T1 line may not have data associated with packet loss,jitter, or latency but may instead only present alarms of red, yellow,or green associated with the perceived performance along a T1 connectionpath. Similarly, there may be a large amount of data associated withwireless networks that are indicative of network performance such assignal to noise ratio, levels of interference, signal strength, or anyother suitable data regarding the performance of the wireless networkthat may be useful and utilized as network performance information.

Continuing with FIG. 3, in addition to generating network performanceinformation based on PIP data packets, the principles of the presentinvention provide for the network nodes 304 to determine real-timetransmission rate or real-time traffic utilization (i.e., a number ofdata packets including real-time content being communicated over networksegments during a time period or, mathematically described, real-timebandwidth use may be determined by tracking the summation of the size ofeach real-time packet that is transmitted in a given time period,collectively. Alternatively, tracking the real-time packets transmissionrate=number of real-time packets*average packet size/given time period).Real-time content is data produced by applications that use real-timeand near real-time data packet communications (e.g., VoIP telephonecalls). Data packets including real-time content (i.e., real-time datapackets) may include payload data 308 representative of speech during atelephone call, video data stream during a live event, music streamduring a live concert or live radio broadcast, or gaming data, possiblyincluding embedded voice, during a competitive live gaming session, forexample. Non-real-time data packets may include payload datarepresentative of content that does not need to be communicated inreal-time (e.g., music download, webpage content, program updatedownload, etc.). Total bandwidth transmission rate or total transmissionrate may also be determined so that if the real-time transmission rateis known, then the non-real-time transmission rate is also known.

Determining bandwidth usage, both real-time and total bandwidth usage,can be accomplished by tracking either individual data packets andpacket flows or internal network element traffic statistics. Collectingthe amount of real-time or total bandwidth usage may be performed in anumber of ways, including examining a priority bit marking (‘P’ bit),type of service (TOS) bit marking, virtual local area network Class ofService (COS) marking, IP address and/or port. Additionally, probes,queuing, scheduler, bus, or path metrics may also be used with any otherinformation associated with a data packet that is capable of indicatingwhether one or more data packets are real-time or non-real-time may beused in collecting real-time and non-real-time bandwidth usage. Forexample, accessing other in-use protocol stacks via probes or “crossstack” communication can provide information from real-time controlprotocols, such as Real Time Protocol (RTP) and Real Time ControlProtocol (RTCP). Real-time protocol packets may be used to identifyreal-time bandwidth rate of communication sessions data packets. Bydetermining bandwidth of real-time and total data packets, andoptionally other PIP information, a call control manager 310 may managenetwork communications sessions in a more intelligent manner.Determining transmission rates may be performed at Layer 1 or Layer 2 ofthe OSI model. However, it should be understood that determining thenetwork performance information may be performed on a different layer ifinformation is available on the different layers to provide enoughinformation to determine bandwidth utilization of real-time and totaldata packets being communicated over a node segment. These segments maybe a shared path resource as in a media gateway to media gateway path,pin-hole firewall access node path, or they could be to a singlesubscriber end-point or intermediate trunking point. It is understoodthat multiple communications devices share the same transmission pathand no single session controller may have knowledge of the in-usereal-time data packet counts or bandwidth state without this informationbeing derived from the use of PIP packets.

Continuing with FIG. 3, the transmission performance collection units306 may include one or more modules to determine the network performanceinformation, both with respect to the communication of real-time contentand non-real-time content, number of real-time sessions, packet lossrate, jitter, delay, etc. The module(s) may be in the form of softwareexecuted by one or more processors, hardware (e.g., ASIC chips),external probes, firmware, or a combination of hardware and software, asunderstood in the art. The modules may be configured to count the numberand size of total data packets and bandwidth of real-time data packetsand non-real-time data packets that are being communicated over a nodesegment (i.e., one or more communications links between two networknodes or connections, processes, or components within a network node),also referred to herein as a network segment. A communications path mayinclude one or more node segments. Counts may include data packets andreal-time packets with and/or without error rate. It should beunderstood that counting non-real-time data packets is equivalent tocounting real-time data packets and total bandwidth because real-timedata packets may be determined by subtracting non-real-time data packetsfrom total data packets and bandwidth. In one embodiment, multiplechannels may be utilized to communicate real-time data packets andnon-real-time data packets along different paths through one or moredevices and communications paths. Channels may include virtual orphysical paths constructed of ports, buses, schedulers, shift registers,cards, and chips used to transfer or move packets through the device.Real-time packet flows may be separated by assigning ports, markings,size, type, and/or other sorting and scheduling methods to map specifictraffic to a specific route or path. Using different channels forreal-time and non-real-time data packets may enable counting real-timedata packets and non-real-time data packets faster than having toanalyze information contained within the data packets (e.g., P-bit indata packet header). Alternatively, real-time and non-real-time portsmay be configured at a network node to monitor and measure real-time andnon-real-time data packets, transmission times, or given path orresource utilization. Real-time and non-real-time bandwidth utilizationcan also be measured by the amount of time the resource is not in use bymultiplying the transmission rate of the path or resource. In additionto measuring the amount of real-time and non-real-time bandwidthutilization, a second measurement to characterize the burst nature(burstiness) of the data flows may be performed. When multiple packetflows of different packetization rates and different bandwidthutilization rates are combined, an average and peak utilization occurs.One example of a measurement to characterize the burstiness of combinedreal-time flows includes using standard deviation of the peak from theaverage calculation. Other mathematical methods may be applied tocharacterize this ability to over-subscribe the real-time flows based onfluctuation in real-time bandwidth usage during the sampling window thatcalculates average bandwidth use. This added measure of burstiness canbe used optionally with the real-time bandwidth usage. Because PIP datapackets 302 effectively operate as a clock on an asynchronous network,the transmission performance collection units 306 may monitor and/orinspect the PIP data packets 302 to determine rates of total datapackets and bandwidth, real-time data packets and bandwidth, andnon-real-time data packets being communicated over a network segment.

FIG. 4 is an illustration of an exemplary data packet stream 400including PIP data packets 402 and data packets 404 a-404 n(collectively 404), the latter including a packet payload of eitherreal-time or non-real-time content. Each of the data packets 404 mayinclude a header portion including a destination address 406 a andorigination address 406 b, among other header information, and a contentor packet payload portion 406 c that includes either the real-time ornon-real-time data along with other transmission characteristics.Although only a single data packet 404 a is shown between successive PIPdata packets 402 a and 402 b, as understood in the art, there may bemany data packets 404 that are communicated between successive PIP datapackets. By determining a total number of data packets and packet sizeboth real-time and non-real-time, communicated during a time duration(e.g., 1 second) and a number of real-time data packets communicatedduring that time duration, bandwidth of the total number of data packetsand real-time data packets communicated over a node link may bedetermined. Alternatively, the amount of time a communications path orresource is not in a transmission state may be utilized to determinebandwidth and data packets communicated. Additional information, such asthe distribution, burstiness, or timing of the real-time flows, may alsobe made available within the PIP packets 402. Together, the real-timeand non-real-time packets may be used in conjunction with the linkcapacity to calculate the average utilization over the interval. Thebandwidth determination may be performed by monitoring the PIP datapackets 402 that are collected to indicate that the time period hascompleted or inspecting the timestamps contained within each of the PIPdata packets 402, which may be more accurate. Monitoring the PIP packets402 may include monitoring one or more PIP packets 402. Performancecalculation modules may track utilization and performance ofcommunications paths and node segments, and create historicalperformance and utilization measure logs. Collected performanceinformation may be used to detect threshold crossings to be communicatedto session controllers, as further described herein. Other networkperformance information may be determined by monitoring the PIP datapackets 402, including jitter, packet loss, and delay, as the PIP datapackets 402 may include information indicative of time sent and countervalue indicative of number of data packets sent between the previous andcurrent PIP packet. The network performance information may further becategorized as real-time, non-real-time, and/or total networkperformance information (see TABLE I). In one embodiment, intermediatelevels may also be established, such as near-real-time, higher prioritynon-real-time, etc.

FIG. 5 is a block diagram of an exemplary network node 500 configured toperform functionality in accordance with the principles of the presentinvention. The network node may include a processing unit 502 thatincludes one or more processors that execute software 504. In oneembodiment, the software 504 may include module(s) that operate as atransmission performance collection function to collect networkperformance information. The processors 502 may be in communication witha memory 506 configured to store information, such as networkperformance information, in registers or one or more tables in memory,as understood in the art. The processors 502 may further be incommunication with an input/output (I/O) unit 508 that is configured tocommunicate over one or more communications networks. I/O unit 508 mayinclude one or more access ports (not shown). The processors 502 mayalso be in communication with a storage unit that is configured to storeone or more data repositories (e.g., databases) that store networkperformance information. The storage unit may work in conjunction withthe memory 506. These memory registers are sometimes referred to asbins. The network node 500 may be one of a wide variety of networknodes, including the Maintenance-End-Points (MEPs) andMaintenance-Intermediate-Points (MIPs) of a Maintenance Entity Group(MEG). The MEPs may include access node devices, such as a digitalsubscriber line (DSL) modem, or Cable Modem and its corresponding Accessnode DSLAM or Cable Management Termination System (CMTS). Mobile dataelement, SIP phone, Video On Demand (VOD) Server or a Media Gateway (MG)device, and/or network-to-network interface (NNI), for example.Additionally, the MEPs may include user network interfaces integratedaccess devices (IADs), session initiation protocol (SIP) devices, orother end-user devices or customer premises equipment (CPE). The MIPsmay include bridges, switches, and routers, for example.

In one embodiment, the memory 506 may store network performanceinformation in bins over a short period of time, such as seconds orminutes, and the storage unit 510 may store historical networkperformance information for longer periods of time, such as hours, days,weeks, or longer periods of time. By storing recent network performanceinformation, remote network nodes (e.g., call control manager, andresource allocation systems and software) may poll the network node 500for the network performance information and receive the networkperformance information in a relatively short period of time as comparedto the network node 500 having to access the network performanceinformation from the storage unit 510. Periodic updates may be retrievedvia polling, event driven on a regular time basis or during unitinitiation or power off, or trigger driven events may also be utilizedto transmit the network performance information. The network performanceinformation may include network performance information indicative ofdata packets including real-time and non-real-time content. TABLE I isan exemplary table that describes an example of network performanceinformation as associated with real-time and non-real-time data packets.Although not illustrated, such data may be identified for communicationin each direction over a particular node segment.

TABLE I Real-Time and Non-Real-Time Network performance informationReal-Time Characteristic Measure Total/Average Max Packet Count 87734749 Avg Packet Count 852 32833 Active use time .87 30005 Peakednesscharacteristic 128 200 Time Period   1 s   1 s Number of RT sessions 156187 Max Bandwidth (Kbps) 877 34.75 Avg Bandwidth (Kbps) 852 32.83 PacketLoss 37 241 Jitter .004 s .006 s Delay .020 s .028 s

Although the time period is shown as 1 second, it should be understoodthat any time period may be utilized to collect the network performanceinformation. Multiple tables or bins may be used to tabulate differenttime periods, such as 15 minute, 1 hour, 1 day, and so forth, may bemanaged for storing the same, different, or additional networkperformance information and, optionally, over different periods of time.In one embodiment, historical network performance information may bestored in a database to enable a call control manager the ability topredict usage of the network node 500 during an upcoming time period(e.g., next 5 second, next 2 minutes, next day, etc.). For example, ifthe network node is over-subscribed with users who perform real-timedata downloads during the 7 pm-9 pm timeframe, then the call controlmanager may utilize that information to route certain calls to othernetwork nodes that have less utilization during that timeframe. Inanother example, real-time and non-real-time files may be stored on aVideo On Demand (VOD) server, in which case actual real-time useinformation could be used to load balance system requests. Many othercall and network control functions may be employed by knowing thereal-time and total data packet network performance information. Otherstatistical analysis uses of this data are possible. Another nearreal-time use is a graphical presentation of this data in OperatorNetwork Management Systems (NMS).

FIG. 6 is a block diagram of the software 504 of FIG. 5 and exemplarymodules configured to determine and collect network performanceinformation in accordance with the principles of the present invention.In collecting the network performance information, one embodiment of thenetwork node 500 may include IEEE 802.1AG and ITU-T Y.1731 modules 602and 604, respectively, to generate and receive IEEE 802.1AG data packetsand determine network performance information associated therewith. TheITU-T Y.1731 module 604 may be a modified ITU-T Y.1731 function that isconfigured to collect network performance information associated withdata packets containing both real-time and non-real-time content (see,for example, TABLE I). The modified ITU-T Y.1731 module 604 may beconfigured to collect performance information, such as maximum andaverage bandwidth for both real-time and total data packets along withother transmission characteristics that are being received and/ortransmitted from the network node. One or more modules 606 may beconfigured to store and communicate collected transmission performancedata. As described with respect to FIG. 5, the transmission performancedata may be stored in memory and/or storage unit in one or more datarepositories, such as database(s) or table(s). Communication of thecollected transmission performance data may be triggered by eventthreshold crossings or pulled by another network system, network node,Element Management Systems, or call control manager, for example, and beperformed on a routine basis or in response to a poll, audit, or event(e.g., dropping below a transmission quality threshold). In addition,although 802.1AG and ITU-T Y.1731 standards are presented for generatingPIP data packets and collecting network performance information, theprinciples of the present invention may use other standards andprotocols for collecting network performance information for real-timeand total data packets being communicated over a node segment.

FIG. 7 is an illustration of multiple exemplary data packet networks 700a and 700 b (collectively 700) having exemplary network nodes configuredwithin the networks 700 to determine and collect network performanceinformation. The data packet network 700 a includes a media gateway 702,router 704, bridge 706, and network-to-network interface (NNI) 708.Other network nodes, such as a session border controller, switch,firewall, computer, satellite, service point or broadband node gateway(VOD server, IP service point, end-point), CPE (customer premisesequipment), wireless handset, or any other packet service network nodemay be configured in accordance with the principles of the presentinvention. More specifically, each of these network nodes may beconfigured with modules, such as the modules described with respect toFIG. 6, which produce PIP data packets and collect network performanceinformation for both real-time and total data packets communicated overthe network. In one embodiment, a network node, such as router 704, maycollect and communicate the network performance information in the formof data packets 710 via communication link 712 to a call control manager714. The communication of the network performance information may becommunicated to the call control manager 714 in response to a poll fromthe CCM 714, periodically, or in response to an event (e.g., packet lossdropping below a predetermined percentage). Because the CCM 714communicates with multiple network nodes, the CCM 714 may be configuredto route calls based on the network performance information collectedfrom the network nodes.

The PIP data packets that are communicated present an opportunity fornetwork performance information to be communicated along a network path.In one embodiment, network performance information may be collected andappended to or otherwise inserted into PIP data packets so that othernetwork nodes can monitor performance along a virtual path. For example,the network performance information may be collected by a network node,stored in a database, and a summary may be appended with a PIP datapacket and communicated to another network node to which the PIP datapacket is communicated. This concatenation process may be performed atregular intervals such as every 5 minutes, hourly, or daily, to minimizethe amount of data communicated over the data packet network and storedat network nodes. The CCM 714 may collect and store historical networkperformance information and utilize such information to monitor trendsin the network 700 a and automatically alter network operation or enablenetwork operators to reconfigure the network. For example, if it isdetermined that the paths to a network node or the network node itselfis not operating properly, the CCM 714 may choose or establish a virtualpath through different network nodes or different paths through the samenode than would otherwise not have been established. As another example,if data packets are being lost, the CCM may choose to force existing andnew call sessions to use CODECs of a lower compression rate on the nodesegment to alleviate congestion and improve call connectivity.

FIGS. 8-9 are block diagrams that more specifically describe structuraland functional operations of network communications devices, (i) end ormid-point devices (FIG. 8), (ii) firewall device (FIG. 9), and head-enddevice (FIG. 10).

FIG. 8 is a block diagram of an exemplary end or mid-point device ornetwork communications device 800 showing structural and functionaloperations used for employing the principles of the present invention.The device 800 includes a network packet port or end-point IP trunk 802that is configured to transmit and receive real-time traffic, such asVoIP, video, RTCP and other real-time data packets 804, via one or moredata ports 806 a-806 c (collectively 806). Each of the data ports 806may have one or more communications lines 808 a-808 c (collectively808). Network counters 810 may operate at the network packet port 802 tocount data packets including real-time content and total data packets.It should be understood that the network packet port 802 may receivedata packets including both real-time and non-real-time content and thatdata packets containing real-time content should not be delayed any morethan is proper to allow for real-time communications (e.g., telephonecalls) to occur without an end-user noticing the delay. A remotemonitoring function 812 that is associated with the network packet port802 enables various network monitors and console systems to exchangenetwork-monitoring data, in the exemplary case, 802.1AG data. A set ofnetwork-side counter functions 814, which may be executed on one or moreprocessors or be a hardware device (modules), operate to count anddetermine a number of real-time data packets and total data packets(i.e., number of data packet having real-time and non-real-timecontent), and determine bandwidth for the data packets includingreal-time content and bandwidth of total data packets over a timeperiod. The network side-counter functions 814 may be performed on eachport, line, and/or entity and generate the network performanceinformation (e.g., packet count and bandwidth) for both transmitted andreceived data packets. Performance collection engines are understood tobe located at the same end-point 812 that transmits the PIP informationto the far-end. However, the performance collection engines receive aPIP stream from the far-end for the purpose of measurement andcollection.

A call control module 816 connected between the network packet port 802and a plain old telephone system (POTS) port 818 on a line side of thenetwork communications device 800 may operate to control and manage callconnections for a network communications device 800. The call controlmodule 816 may include one or more CODECs using real-time transportprotocol (RTP) and/or real time control protocol (RTCP). Thenetwork-side counter functions 814 may be in communication with the callcontrol module 816 and generate packet count and bandwidth informationbased on the data packets being handled by the call control module 816.As understood in the art, CODEC stacks may be any session controlprotocol based or operate in-band, such as Internet Group ManagementProtocol (IMGP) for video multicasting, and may have knowledge of thetype of call being set-up. Such a configuration also applies to pin-holefirewalls, media gateways, call controllers, and bandwidth reservationprotocol systems that use control stacks, bandwidth reservation orresource reservation functionality. It is understood that a network-sidecounter function may also contain a pseudo utilization counter thatindicates bandwidth reservation use information to be passed along withreal bandwidth utilization performance and utilization information. Thepseudo or reserved use provides an indication of bandwidth use for callsthat are being setup, on hold, or simply being reserved. The reservationinformation may be used in providing the session controllers with theinformation collected or determined at the network segment. The CODECstacks and real-time protocols can be used to track correlated persession bandwidth use and report total use to the network side counterfunction. A few example techniques for counting data packets anddetermining bandwidth of data packets including real-time data packetsand total data packets are provided below:

-   -   (i) bandwidth use via reading CODEC settings and packetization        rates, (a) in use, and (b) reserved    -   (ii) if IGMP is used for video, (a) track total number of        streams—real time, and (b) track individual stream use        (bandwidth varies) via CODEC and packetization rates.

Additionally, any network EMS or protocol stack provisioning engine maycommunicate with the counters 814 and 826 so as to monitor call shapingat the network communications device 800. Real-time performance enginesand probes may also communicate with control stacks. Client interactingcontrol stacks may enable a user to select functions to perform in userdevices. In summary, these applications may be used on (i) virtual,logical, and physical ports, (ii) firewalls, (iii) traffic shapinginstances, (iv) service agents, (v) network elements, and elementschedulers, (vi) and/or service points, such as gateways, VOD servers,and conference bridges for example.

The network communications device 800 may further include one or moreline-side packet ports 820 a-820 n (collectively 820) that areconfigured to communicate data packets. The line-side packet ports 820may include respective remote monitors (RMONs) 822 a-822 n (collectively822), PIP generators, and performance collection elements (not shown),that are used for exchanging network monitoring data with other networkdevices and counters 824 a-824 n (collectively 824). Similar to thenetwork side, the line-side may include a set of line-side counterfunctions 826 that count packets and determine bandwidth of data packetscontaining real-time content and total data packets. A call controlmodule, CODEC stack, probe/sniffer interface 828 may be connected to anetwork-side port and line-side port. The call control module probe 828may be in communication with both the network-side counter functions 814and line-side counter functions 826 to count data packets and determinebandwidth of data packets being communicated between a network-side portand line-side port.

FIG. 9 is a block diagram of an exemplary pin-hole firewall device 900showing structural and functional operations used for employing theprinciples of the present invention. The pin-hole firewall device 900includes multiple physical and logical ports 902 a and 902 b(collectively 902) through which data packet communication sessions arepassed via a shared trunk or Ethernet Virtual Circuits (EVCs) 904 a and904 b (collectively 904). The data packets (not shown) communicated overthe EVCs 904 are communicated through counter and computation functions906 a and 906 b (collectively 906) that are configured to count datapackets including real-time and non-real-time content and determinebandwidth associated with each. In determining bandwidth, PIP datapackets, may be used as a clock. The counter and computation functions906 may be disposed prior to a pin-hole firewall function 908 thatoperates by allowing an application to take control of a port during acommunication session. The counter and computation functions 906 haverespective flow counters 910 a and 910 b (collectively 910) that operateto individually count real-time data packets and total data packets. Inaddition, the flow counters 910 may perform the computations forbandwidth for each port. A call control module probe/sniffer interface912 may be in communication with the flow counters 910 and pin-holefirewall function 908 to provide input into the flow counters 910 basedon call control protocols 914 a and 914 b, respectively. The counterfunctions store the utilization and performance management informationas a measured resource for both the network element itself, and thesession controllers with which the pin-hole firewall may communicate.Communication of the performance and utilization information can be withbut not limited to, a session controller, bandwidth reservation system,network management system, higher layer IP protocols, and othercommunication systems and systems software. Further description of theoperation of collecting and communicating network performanceinformation from the pin-hole firewall device 900 is provided in FIG.11.

FIG. 10 is a block diagram of an exemplary access node device 1000showing structural and functional operations used for employing theprinciples of the present invention. The access node device 1000 may beconfigured as a head-end device (e.g., video distribution system), cablemodem termination system (CMTS), digital subscriber line accessmultiplexor (DSLAM), or digital line concentrator (DLC), for example.The access node device 1000 may include a network port 1002 and accessports 1004 a-1004 n (collectively 1004) that are connected tocommunication lines 1006 a-1006 n (collectively 1006). VoIP, video, andRTCP or other real-time data packets, for example, may be communicatedover the access transmission paths or communications lines 1006. Remotemonitors 1008 a-1008 n are respectively associated with each of theaccess ports 1004 and may operate to receive 802.1AG data packets orotherwise and include PIP packets and performance collection-functions.Also associated with the access ports 104, counters 1010 a-1010 b(collectively 1010) may be configured to count total data packets anddata packets containing real-time content being received via respectivecommunication lines 1006. It is understood that some of the real-timeinformation flows at this node may be available via control protocolimplementations, such as Video IGMP, in which case the control protocolstack or other counter mechanisms can be utilized to measure and reportreal-time traffic to the PIP generators. It is also understood thatother real-time streams may exist that are not under the control of aprotocols stack at this node in which case the real-time and totalbandwidth characteristics may be measured via methods described herein.A remote monitor 1012, which may include a PIP packet generator andperformance collector, may be associated with network port 1002 and beused to generate and communicate 802.1AG data packets having networkperformance information generated by a statistics engine 1014. Thestatistics engine 1014 may be configured to generate statisticsassociated with the data being communicated from the access ports 1004to a communications network (not shown) via the network port 1002. Thestatistics may include raw data and mathematically computed data,including (i) real-time bandwidth, (ii) total bandwidth, (iii)provisioned bandwidth, (iv) historical real-time bandwidth, and (v)averages, such as average real-time bandwidth and average totalbandwidth. Indicia representative of the statistics may be communicatedto the remote monitor 1012 and communicated to other network devices towhich the access node 1000 is in communication by appending the indiciato PIP data packets, for example, or otherwise communicate thestatistics in separate data packets. The indicia may be in any form,such as XML, of a communication protocol being communicated from theRMON 1012 and/or network port 1002. Threshold crossings may triggerspecial messages to other systems, as well as systems polling themeasured resource pool inside the statistics engine 1014. Furtherdescription of the operation of collecting and communicating networkperformance information of the access node device 1000 is provided inFIG. 11.

A management function 1016 may be in communication with the statisticsengine 1014 and include a number of management functions, including, butnot limited to, resource reservation protocol (RSVP), call admissioncontrol (CAC), and provisioning. Session controllers 1018, which may beexternal from the access node 1000, may be a resource access controlsystem or facility (RACS or RACF) that operates as a security system andprovides access control and auditing functionality. The sessioncontrollers 1018 may communicate with the management function 1016 tomonitor operation of the access node 1000 by collecting the statisticsgenerated by the statistics engine 1014.

FIG. 11 is a block diagram of exemplary modules 1100 configured todetermine network performance information associated with data packetscommunicated with network communication devices described in FIGS. 8-10.The modules 1100 may be configured as software, hardware, firmware, orcombination thereof and be generalized into the standards for differentnetwork communications devices to ensure consistency throughout one ormore networks. The modules may include a probe module 1102, CODEC stackmodule 1104, statistics module 1106, and statistics modified operations,administration, and management (OAM) module 1108. The probe module 1102operates using counters to count various types of data packets,including (a) total packets and/or total bandwidth, (b) marked packets,(c) packets provided special bandwidth treatment, and (d) real-timeprotocol flows (e) or other. The marked packets may be counted based onspecifics within the packet such as (i) specific type of service (ToS)level markings contained in the IP header, (ii) specific packets in anEthernet virtual channel (EVC) or Class of Service markings (COS), (iii)P-bit marking used in 802.1Q tags, (iv) differentiated services(Diffserv) field (IP header), or any other suitable basis. The packetsprovided special bandwidth treatment may include: (i) specific packetsin high priority schedules (hardware specific), and (ii) being treatedby a QoS engine (such as Diffserv). The investigation of real-timeprotocol flows may include (i) RTP & RTCP reading (observing the realuse in the header), and (ii) other in-band protocols that can be read interms of use (e.g., CODEC, profile, bandwidth, call reservation, etc.).The counters may be pushed, polled or be accessed by the statisticsmodule 1106.

The CODEC stack module 1104 may include real-time counters and beinfluenced by operation of the CODEC. Real-time data packet counters maycount when the CODEC stack is (a) in use and/or (b) reserved or (c)other. Counters may also track IGMP use for video to count (a) number ofreal-time streams and/or (b) individual stream use, where the bandwidthmay vary. Further, traffic shaping by a provisioning engine may befactored by the CODEC stack module. Real-time performance engines thatinterwork with control stacks may also affect how the count of the CODECstack module operates to collect network performance information. Clientinteracting control stacks that allow the user to choose functions tocontrol specific applications may be tracked.

The statistics module 1106 may operate to collect statistics orperformance information of data packets including real-time content(e.g., VoIP data stream) and total data packets communicated to and fromthe network communications device. In one embodiment, the statisticsmodule 1106 is a modified Y.1731 application. The statistics module 1106may operate to collect (1) transmitted statistics, (2) receivedstatistics, and (3) individual counters for each port, line, or entity.The statistics module may perform mathematical calculation on thecollected information. The transmitted statistics may determine, forexample: (a) total bandwidth based on all the data packets that arecommunicated over a time period, where the time period may be determinedfrom PIP data packets generated by the network communications device,(b) bandwidth of the real-time data packets communicated during the timeperiod, (c) real-time data packets counted, and (d) other collected orprocessed information. The received data packet statistics may includethe same as the transmitted statistics, including (a) total bandwidth ofall the data packets counted, (b) real-time bandwidth of the real-timedata packets, and (c) real-time data packets counted, and/or (d) othercollected or processed information. It should be understood that otherstatistics, including average values, maximum values, trended predictionvalues, or other suitable measurements or calculations that could bebeneficial to other network communications devices or soft-switches(e.g., call control manager) may be collected.

The modified OAM module may include any of the following functions: (1)sequence counting to ensure that data packets are being received in theproper sequence, (2) appending a sequence flag to 802.1AG packets, (3)adding a name for a circuit leg or segment, (4) including indicia indata packets to identify a carrier collecting network performanceinformation, and (5) communicating collected transmission performancestatistics. The communication of the collected network performancestatistics may be performed in response to a poll from another networkdevice (e.g., call control manager), in response to an event, orperiodically. Such communication may be accomplished by communicatingthe network performance information or statistics by appending orotherwise including the information with other data packets. Forexample, the network performance information may be added to payloads ofdata packets that are being communicated to other network nodes.

FIG. 12 is an illustration of an exemplary process 1200 performed onnetwork nodes in a data packet network. The process 1200 starts at step1202, where data packets containing real-time content (e.g., from atelephone call) being communicated over a network node segment of apacket network are monitored. Monitoring the data packets may includedetermining that the data packets include real-time content in thepayload by examining the header for the P-bit or otherwise determiningthat the content includes real-time content. At step 1204, at least oneitem of network performance information associated with data packetscontaining real-time content communicated over the network node segmentof the packet network may be determined. The network performance mayinclude real-time bandwidth usage, total bandwidth usage, packet loss,delay, and/or jitter, for example. While such network performanceinformation is typically used for determining operation of a nodesegment, other performance characteristics, such as packet rate andbandwidth, may also be collected for real-time and total data packets.At step 1206, indicia representative of the at least one item of networkperformance information regarding the communicated data packetscontaining real-time content may be communicated to a network element,such as a soft switch or call control manager. The indicia may be in theform of a Call Control protocol, SNMP message, XML protocol, HTMLprotocol, or any other protocol utilized by a data packet of acommunications standard.

Distributed Transmission Performance Tables in Network Nodes

FIG. 13 is an illustration of an exemplary network node 1300 configuredto perform functionality and communications over a packet network inaccordance with the principles of the present invention. The networknode or network communications device 1300 may include a processor 1302that executes software 1304 to perform operations for the network node1300. The network node 1300 may be a router, switch, media gateway, orother network communications device, and include software that performsany function associated with typical operations of a network node. Theprocessor 1302 may be in communication with the memory 1306. The memory1306 may store a table that includes network performance informationassociated with node segments over which the network node 1300communicates. The processor 1302 may further be in communication with anI/O unit 1310 and storage unit 1312. The I/O unit 1310 may be utilizedto communicate data packets 1314, including content data packets 1314 aand PIP data packets 1314 b over node segments 1316 a-1316 n(collectively 1316) to other network nodes 1318 a-1318 n (collectively1318).

In one embodiment, the PIP data packets 1314 b may include networkperformance information of respective network communications devices1318 describing transmission performance over respective node segments1316. In addition, each of the network communications devices 1318 maystore the network performance information describing the node segmentsto which each is in communication (e.g., network node 1300). The networkperformance information may be communicated to the network node 1300 ineach PIP data packet 1314 b, periodically (e.g., every 100^(th) PIP datapacket, once per second, every 5 minutes, etc.), in response to an event(e.g., in response to a network performance information value crossing athreshold value), or in response to a poll or request from the networknode 1300, for example. The quantity and types of performanceinformation contained in each PIP packet could vary between successivecommunications. For example, derived or summarized information may becommunicated on five minute intervals, and other information may becommunicated in other intervals.

As previously described, the software 1304 may be configured to generatenetwork performance information associated with node segments betweennetwork nodes. The table 1308 may include network performanceinformation associated with the network node 1300 and network nodes 1318with which the network node 1300 communicates. Although the table isshown to be stored in memory 1308, it should be understood that thetable 1308 may be stored in storage unit 1312 in that the memory andstorage unit, for the purposes of the principles of the presentinvention, are both considered to be memory. It should further beunderstood that the term table 1308 is generally descriptive of datastored in a defined arrangement of fields and is descriptive of anyorganized data set, such as a database or data file containing datafields. The term table is also inclusive of multiple tables that areassociated with one another.

TABLE II provides an exemplary table including network performanceinformation. The table may include segment numbers or otheralpha-numeric indicia, name of associated segment, and networkperformance information in one or both directions for each networksegment or path (e.g., east-to-west, west-to-east). The networkperformance information may include additional and/or other informationrepresentative of transmission characteristics (e.g., transmission rateand bandwidth) along each bearer path. Although not shown, the networkperformance information associated with data packets including real-timeand non-real-time content (see, for example, TABLE IV). TABLE II mayalso include node segments associated with the internal performance of anetwork node or network device. For example, if a particular networknode is a network switch, the operation of such network switch mayimpact network performance and thereby may have its own networkperformance information. For example, a network node, such as a switchor router, may itself cause packet loss or introduce a delay in thedelivery of packets. Thus, identifying particular elements or processeswithin a network node may be useful in monitoring, reporting,compensating for, troubleshooting, or otherwise reacting to problems innetwork performance. More particularly as illustrated in TABLE II,specific buffers or queues may be identified such as buffer/queue Acorresponding to segment number 5. For example, a particular bufferwithin a network device may overflow resulting in lost data packets atthat node and correspondingly an underflow event at a downstream device.Similarly, a particular processor B within a network device isillustrated in TABLE II as being associated with segment number 6. Forexample, a processor may not be able to keep up with processes requiredto switch or route packets over a particular network. In fact, manyentities are experiencing problems with processor performance asnetworks become more utilized. Likewise, the internal switching fabricof a particular network device such as a switch or router may alsoimpact network performance as data packets are required to be switchedor routed by such device. Such a fabric is illustrated in TABLE II asbeing associated with segment 7. For example, both packet loss and delaycan be introduced by the performance of a switch.

Although not illustrated herein, other software, processes, processors,memory components, or any other component of a particular network nodeor device that may impact network performance may be included in atable, such as TABLE II, based on data associated with the performanceof such components. Likewise, although not illustrated in TABLE II, manyother types of network performance information may be included relativeto any particular node segment. For example, a wireless switch on awireless network may have upwards of twenty or thirty factors thatinfluence network performance. For example, interference,signal-to-noise ratio, signal strength, and battery or power level mayall affect network performance and may all be represented in a tablesuch as the one illustrated in TABLE II. Likewise, switches, hubs,bridges, or other interfaces between networks, portions of a network orvarious media of communications may instead or additionally storeinformation such as alarms, notifications, signal characteristics, orany other suitable type of data capable of being used to evaluatenetwork performance. For example, with regard to a simple T1 connection,the only information available regarding such connection may be theexistence of a red, yellow, or green indication or alarm. Likewise, aDSLAM device may have very different information relevant to networkperformance than the information available from a core IP router. All ofthe foregoing information is considered network performance informationfor purposes of this application and may be incorporated in any table,bin, database, or PIP packet described herein.

TABLE II Network Segment Status Table Network Segment Status Table Eastto West West to East Packet Packet Segment # Name Jitter Delay LossJitter Delay Loss 1 Bearer Path 1 2 Bearer Path 2 3 Bearer Path 3 4Bearer Path 4 5 Buffer/ Queue A 6 Processor B 7 Fabric C

TABLE III is another exemplary network segment status table thatincludes counter and timestamp. The counter and timestamp may be used todetermine the network performance information. For example, thereceiving node (e.g., node 1318 a) may use the counter to determine thetotal number of data packets that were communicated over the nodesegment or network segment. If, for example, the counter indicates that200 data packets were communicated since the previous PIP data packet,then the receiving node may determine how many data packets werereceived to determine if any data packets were lost. For example, if thecounter indicates that 200 data packets were communicated and thereceiving node determines that 182 data packets were received, then 18data packets were lost. Also, the delay may be determined by recordingthe time that the PIP data packet is received and subtracting it fromthe PIP data packet timestamp.

TABLE III Network Segment Status Table Node Packet PIP Data PacketTimestamp Segment Dir. Node 1 Node 2 Delay Jitter Loss Counter(hh.mm.ss.dd.mm.yy) 1316a East 1300 1318a .04 .002 23 23472303.18.43.12.07.07 1316a East 1300 1318a .05 .003 18 23492303.18.44.12.07.07 1316n East 1300 1318n .03 .001 3 7483203.22.17.12.07.07 1316n East 1300 1318n .06 .002 42 7583203.22.18.12.07.07

The network performance information shown in TABLES II and III arerepresentative of PIP data packets associated with unidentified contenttype. However, in accordance with the principles of the presentinvention, the network performance information may be determined withrespect to communications of data packets including real-time contentand non-real-time content. TABLE IV includes network performanceinformation that distinguishes real-time content and total content(i.e., real-time and non-real-time). Although the direction is shown asgoing east in both of TABLES III and IV, it should be understood thatwest direction network performance information may also be included inthe table. By providing network performance information specificallyrelated to real-time data packets, an understanding of how the packetnetwork is operating for different content types can be determined. Inaddition, these tables can provide network performance information forone-way, bi-directional unicast or multicast traffic flows.

TABLE IV Network Segment Status Table - Total and Real-Time Total RT PIPData Packet Node Total Total Total Packet RT RT RT Packet RT TimestampSegment Dir. BW Delay Jitter Loss BW Delay Jitter Loss Counter(hh.mm.ss.dd.mm.yy) 1316a East 1.54 .04 .002 23 1.25 .03 .002 12 23472303.18.43.12.07.07 1316a East 1.62 .05 .003 18 1.27 .07 .001 9 23492303.18.44.12.07.07 1316n East 1.52 .03 .001 3 0.7 .05 .002 3 7483203.22.17.12.07.07 1316n East 2.25 .06 .002 42 1.85 .04 .002 40 7583203.22.18.12.07.07

Continuing with FIG. 13, the software 1304 executed by the processor1302 of the network node 1300 may examine the network performanceinformation in the table and determine whether any of the networkperformance information parameters cross a threshold value. For example,one or more threshold values may be established for real-time bandwidthuse or real-time packet loss or total packet loss. Thresholds couldinclude a derived ‘watermark’ such as a condition that warrants networkoperator consideration, but is not critical (e.g., a ‘yellow’ alarmcondition) or ‘watermarks’ indicating a moving peak during a definedtime-window. If the software determines that the packet loss or otherdata included in the network performance information associated with anode segment crosses above the threshold value, then a call controlmanager module may be notified to change a network component (e.g., slowdown a CODEC) or re-route current and/or future calls from that nodesegment. Alternatively, if it is determined that the total bandwidth ishigh while the real-time bandwidth is also high (e.g., node segment 1316n at time 03.22.18.12.07.07), then the call control manager may initiateprocesses to slow down or halt the set up of new data streams containingnon-real-time or real-time content until the real-time content demanddecreases or initiate a disconnect based upon some criteria, such apriority bit markings or otherwise. A call control manager or individualnode may alter a device (e.g., slow down a CODEC) or communications(e.g., change modulation) to attempt to improve transmissionperformance. It should be understood that other applications may bederived from monitoring the network performance information contained inthe table(s).

FIG. 14 is a flow chart of an exemplary process 1400 for managingnetwork communications. The process 1400 may start at step 1402 bycommunicating first data packets via at least two node segments on apacket network to at least two network communications devices. Seconddata packets communicated from the network communications devices viarespective node segments may be received at step 1404. The second datapackets may include network performance information generated by thenetwork communications devices in response to receiving the first datapackets. The second data packets may be utilized to exchange transmittedand received performance information between network segment end-points.In one embodiment, the first data packets are PIP data packets. Inaddition, in one embodiment, the second data packets are PIP datapackets. At step 1406, a table containing network performanceinformation associated with the node segments over which the second datapackets are communicated may be stored. At step 1408, the second datapackets may be parsed to access the network performance information,wherein parsing includes reading the content contained in the seconddata packets. In one embodiment, field identifiers included in thepacket may define the start and end of a transmission performanceparameter and may be read to access the value of the transmissionperformance parameter. At step 1410, the network performance informationmay be stored in the table. Communications over the node segments may bealtered based on the stored network performance information at step1412. The communications may be current or future communications.

Peer-to-Peer Distributed Call Control Using Distributed Tables

The network performance information that is stored in tables at eachnode may be used by the nodes to make network control or routingdecisions when placing or routing calls. These decisions may be based onthe network performance information that is indicative of networkperformance at a node segment associated with a node or other nodesegments for which the node has access in tables stored therein. Thedecisions may include employing congestion avoidance processes asunderstood in the art. Routing decisions, such as routing a call via adifferent transmission path to a called party, may be performed at thenode to avoid congestion or other transmission problem at a nodesegment. Still yet, the node may determine that packet loss is high sothe node may negotiate a lower CODEC bandwidth for on-going, and newsessions with another node prior to or during a telephone call in aneffort to minimize packet loss between the nodes. It should beunderstood that the node making the decisions may include customerpremise equipment, such as a SIP telephone, or any other node within apacket network, including wireless access points, DSL modems, and/orcable modem devices that suffer variable bandwidth availability. Thedistributed call control, in essence, may include the same or similarfunctionality as may be performed by a call control manager.

Centralized Network Performance Information Table

FIG. 15 is an illustration of an exemplary packet network 1500 having acall control manager 1502 with a centralized table 1504 of networkperformance information for use in managing call communications over thepacket network 1500. The CCM 1502 may include the same or similarhardware as provided in FIG. 5, and execute software configured toperform call control operations for end-users on the packet network1500. The centralized table 1504 may include network performanceinformation generated by network communications devices (e.g.,end-points and intermediate points) operating on the packet network 1500that indicate operation of the node segments. For example, router orswitch 1506 may collect network performance information for nodesegments 1508 a-1508 f (collectively 1508). As previously described, thenetwork performance information may be generated through the use of PIPdata packets being communicated over the node segments. It should beunderstood that node segments may traverse from end-to-end and includeintermediate points so that an overall packet network communicationspath can be described. This path could contain one or more communicationtechnologies and protocols, such as Ethernet, SONET, IP, and ATM. Forexample, a node segment may extend from network-to-network interface orsession border controller 1510 to end-user 1512 to describe transmissionperformance for bearer paths 1508 d, 1508 a, and 1508 e and networkcommunications devices between the end-points, including router 1506 andnetwork access node 1514. Generation of the performance information mayuse standard or modified protocols, such as the IEEE 802.1AG protocol,to generate the information as associated with the PIP data packets. Theperformance information may be gathered and stored using a modifiedstandard protocol, such as Y.1731, to include data packets containingreal-time content and total data packets (i.e., real-time andnon-real-time content) so that the CCM 1502 may make call managementdecisions based on the type of calls or sessions that are being placedor currently operating on the packet network 1500.

Collection of the network performance information may occur at regularintervals (e.g., every second, minute, hour, four hours, day, week,month, or otherwise). Collection of the network performance informationat regular intervals, especially shorter intervals, may add overhead tothe CCM 1502 and network communications devices, so other collectionschemes may be utilized for communication of the network performanceinformation to the CCM, such as event and request driven collectionschemes. Event driven communications of the network performanceinformation may occur if a network communications device (e.g., mediagateway 1516) determines that call quality has degraded below apredetermined threshold. For example, if jitter of real-time datapackets increases above a predetermined threshold value, the networkcommunications device may communicate current network performanceinformation to the CCM 1502 for storage in the centralized table 1504.Alternatively, a message or alert may be communicated to the CCM 1502 tonotify the CCM 1502 of the node segment problem, which may cause the CCM1502 to store a value indicative of a problem to be included in thecentralized table. Request driven communications may be performed by theCCM 1502 to send a poll or request to each of the network communicationsdevices to communicate current and/or historical network performanceinformation generated and/or collected by respective networkcommunications devices. The centralized table 1504 may include the same,similar, and/or additional information as described with respect toTABLES I-IV.

The table 1504 may be used by various algorithms, thresholding events,or processes to determine routing changes, CODEC usage choice, or othercall related functions that coincide with obtaining suitable or the bestcall quality using the network available. As provided in TABLE II, thenetwork performance information may include transmission qualityparameters (e.g., real-time bandwidth, jitter, delay, packet loss) in aduplex fashion (e.g., east to west and west to east). The values storedin the table may include derived data and actual raw data generated atthe network communications devices, scaled data representative of theraw data (e.g., scale between 1 and 10 with 1 being the optimumcapability and 10 being the worst capability or vice-versa), or indicia(e.g., grade rankings A-F) representative of the quality of the rawdata. It should be understood that virtually any captured or deriveddata representative of the network performance information may be storedin the table 1504 that provides the CCM 1502 with the ability to managecalls on the packet network 1500.

In using the network performance information in the centralized table1504, call processing within the CCM 1502 may use the networkperformance information for calls being set up. To accomplish this, theCCM 1502 may determine the route taken for the different calls based onlocation of the end-points within the packet network 1500 of the serviceprovider. Since the CCM 1502 has end-point information recorded inconventional provisioning tables, segment information may be added tothese provisioning tables that would provide information on how thebearer path would traverse the packet network 1500. This informationwould be added to both line and trunk provisioning tables within the CCM1502. TABLES V and VI show possible configurations of conventionalprovisioning tables extended to include segment information as collectedby the CCM 1502 from network communications devices.

TABLE V Line Information Table Line Information Table Existing Line1^(st) 2^(nd) Line Name Line Number Information Seg Seg N^(th) SegEnd-User 1 NPA-NXX-1234 . . . 1 3 N/A End-User 2 NPA-NXX-0987 . . . 2 3N/A . . . . . . . . . . . . . . . . . . End-User N NPA-NXX-0298 . . . 17 N/A

TABLE VI Trunk Group Information Table Trunk Group Information TableExisting Trunk Group Trunk Group 1^(st) Number Location Information Seg2^(nd) Seg N^(th) Seg XXYYYY (Phys Location) . . . 1 3 N/A XXZZZZ (PhysLocation) . . . 2 3 N/A . . . . . . . . . . . . . . . . . .

As shown in TABLES V and VI, network performance information or, asshown, representative values of the actual network performanceinformation for each segment may be stored with the associated linesand/or trunk groups. The segment information may be configured as shownabove with a given set of values that corresponds to the node segments,such as the bearer paths 1508, network communications devices, orcombination thereof, used in the transport of the call through thepacket network 1500. The values could also be provisioned in a vector,using commas to delineate the segments. That is, the segments used incalls to/from the chosen end-point may be shown as X, Y, Z, AA, etc. Nomatter how the tables are provisioned, the network performanceinformation or summary thereof may be utilized by call processingfunctions of the CCM 1502.

More specifically, to provide call control based on the lower layerstatus of the underlying packet network 1500, the CCM 1502 may employ amechanism that queries the segment status table for each call.Algorithms may be used to access raw network performance informationstored in the table and convert the raw data into a value that can beused for call processing. These algorithms may take different forms,such as determining the highest value of the network performanceinformation columns (e.g., east-to-west delay, jitter, packet loss orwest-to-east delay, jitter, real-time packet bandwidth, packet loss (seeTABLE I)) and using the highest value as the status value of thesegment. If using these transmission performance characteristics, threevalue range scales could be enacted that to signify (i) whether thesegment is running normally, (ii) if there is slight impairment, or(iii) if the segment is too congested for added traffic. The three valuerange scale could be set at the CCM 1502 in an overall provisioningtable that is commonly used in soft-switch development. For example, ifthe results of the algorithm are in a 1-10 scale, where 1 is the bestavailability and 10 is the worst availability, the “normal” availabilitycould be a range of 1-3, impaired availability could have a range of4-7, and congested availability could have a range of 8-10.Alternatively, the status may be defined using other indicia, such ascolors (e.g., green, yellow, red) or letters (e.g., A, B, C).

FIG. 16 is a flow chart of an exemplary process 1600 for using networkperformance information stored in a centralized table for controllingcalls by a call control manager. The process starts at step 1602. Atstep 1604, node segment values generated from a call processing modulemay be accessed from a table. In one embodiment, the table iscentralized and located at a call control manager located on a networkand include network performance information for each of the packetnetwork communications devices located on the packet network.Alternatively, the table may be located at a network communicationsdevice located on a packet network. In another embodiment, the table maybe a distributed table, such as those stored on network communicationsdevices and the network performance information may be accessed whenneeded. At step 1606, value range information may be requested from anoffice parameter, where an office parameter determines what values areconsidered “normal,” “impaired,” or “congested.” Value range parametersmay be defined by node segment status value ranges: normal: a-b,impaired: c-d, and congested: e-f, where a-f are integers ranging from 1to 10. At step 1608, the value ranges may be received and a node segmentstatus value may be compared to range values. At step 1612, adetermination may be made if the node segment status value is greaterthan 0 and less than “c” of the “impaired” segment status value range.If so, then at step 1614, a “normal” status is set back to the callprocessing module. The process ends at step 1616. If at step 1612, thenode segment status value is determined to have a value of “c” orgreater, then a determination may be made at step 1618 as to whether thenode segment status value is within the “impaired” range or “congestedrange.” If “impaired,” then an “impaired” status may be sent back to thecall processing module. Otherwise, a “congested” status is sent back tothe call processing module. The process 1600 may be used for each nodesegment over which the CCM may route a call or over which a call iscurrently routed. If an “impaired” or “congested” determination isreturned by the process 1600, the call processing module may elect toselect a different route or re-route an ongoing call or otherwise.

FIG. 17A is a flow chart of a high-level process 1700 for centrallycontrolling communications on a packet network. The process 1700 startsat step 1702 by communicating with multiple network communicationsdevices over a packet network. In one embodiment, a call control manageris communicating with the network communications devices. At step 1704,network performance information of node segments providingcommunications between the network communications devices on the packetnetwork is stored in association with indicia representative of the nodesegments. The indicia may be alpha-numeric and describe a physicallocation or logical address associated with the node segments. It shouldbe understood that the node segments may refer to an individual networkcommunications device for describing transmission path performancethrough the device itself, transmission lines, or combination of devicesand lines. In another embodiment, node segments may be an aggregate oftwo or more segments between two end-points. Network communications bythe network communications devices over the node segments may becontrolled based on the network performance information at step 1706.The network communications may be controlled in a variety of ways,including re-routing calls, changing call routes during a call, and/orchanging operation of network communications devices (e.g., reducingbandwidth of a CODEC).

Packet Network Diagnostics

In addition to being able to re-route calls in the event of adetermination being made of a network node or segment being impaired orcongested, the principles of the present invention provide for packetnetwork diagnostics to be made manually, semi-automatically, orautomatically based on network performance information collected fromone or more network nodes. In one embodiment, network performanceinformation may be collected and a parameter may be monitored. One ormore threshold values may be established for use in determining that thenetwork performance information parameter. For example, an upper andlower threshold may be established to ensure that transmission rate ofdata packets including non-real-time content so that a customer does notreceive higher or lower transmission speed than contracted. In oneembodiment, the network performance information may be collected andmonitored at a central location on a packet network. Alternatively, eachindividual network node may monitor itself and other network nodes onthe packet network and be configured to initiate the diagnostics.

If a determination is made that a threshold is crossed by the parameterbeing monitored, then diagnostics may be initiated. The diagnostics mayinclude a wide range of functions, including initiating a loop-backtest, trace route, modified trace route, ping, or otherwise, asunderstood in the art. Additionally, a command may be issued to anetwork node at which a network performance information parameter thatcrosses a threshold is associated to initiate a diagnostics routine andreturn a result from the diagnostics routine. For example, a softwareroutine may be executed for the network node to execute one or moreself-tests associated with data packet communications at the networknode. It should be understood that the diagnostics may be initiated tomonitor network nodes, segments, gateways, or any other networkcommunications device. In addition, the network performance informationfrom a second packet network owned by another communications carrier maybe monitored by an operator of the packet network and diagnostics may beperformed, but return limited result information to avoid sharingconfidential information, for example.

Still yet, if the network performance information parameter determinedto cross a threshold is associated with communications of data packetsincluding real-time content or non-real-time content, the diagnosticsmay be directed to determining whether a problem exists withcommunications of data packets including real-time or non-real-timecontent individually depending on the problem that exists. Otherdiagnostics as understood in the art may be initiated in response to thetype of network performance information parameter determined to cross athreshold.

Restricting Shared Access to Tables

FIG. 17B is one embodiment of a permission table that may be utilized toestablish permission, state, or access levels by various networkparticipants to network performance information that has been collectedover one or more networks. Such network performance information may bestored in a PIP packet, in a table, bin, or other memory structure at anetwork node or access point, or at a central network or inter networkresource, such as an overall network performance table, or a table usedby a CCM, NOC, or EMS system.

More particularly, a permission table is illustrated as Table 17 b 0.Table 17B0 may include fields associated with an entity identifier 17 b2, a segment identifier 17 b 4, and one or more network performanceinformation identifiers 17 b 6.

Entity identifier 17 b 2 may be an identifier associated with anindividual network participant, such as a subscriber, network operator,VPN provider, or other network participant. Alternatively, entityidentifier 17 b 2 may be an identifier associated with a group orcategory of network participants 17 b 10. More particularly, such anidentifier may identify a class of participants in a network, such as asubscriber group, a network operator group, a VPN provider group, or anyother suitable category. For example, there may be an identifierassociated with the operator of the particular network or networksregarding which network performance information is stored in Table 17 b0 such that administrative personnel, devices, or processes of suchnetwork operator may have full access to all of the information in Table17 b 0. Alternatively, a group of network operators who are theoperators of other networks in communication with the network that isthe subject of the table illustrated as 17 b 0 may be given restricted,i.e., a much lower degree of, access to network performance informationstored in Table 17 b 0.

Segment identifier 17 b 4 may be utilized to identify a particularnetwork segment, such as a connection path between two network elements,a network element itself, or a particular process or component of anetwork element. Thus, an individual network segment may be identifiedas illustrated as 17 b 12 while a category of network segments, such asnetwork-to-network interfaces (NNI), may be identified as illustratedrelative to 17 b 22. Similarly, an entire network may be identifiedcollectively to represent all network segments located within suchnetwork as illustrated relative to 17 b 24. Additionally, a particularnetwork path through the network including all network segments locatedalong such path may be identified such as is illustrated relative to 17b 26. Similarly, a category of paths may be utilized as an identifier,such as, for example, all paths between the CPEs of a particularcustomer or group of customers, may be identified as illustratedrelative to 17 b 28. Likewise, portions of paths including perhaps onlythose network segments between a customer's CPE and a network accesspoint may be identified to give the customer or a network provideraccess to line state information for such customer as illustratedrelative to 17 b 30.

NPI identifier 17 b 6 may identify different categories of networkperformance information that are available to be accessed by aparticular network participant regarding a particular network segment orcategory of either of the foregoing. For example, individual items ofnetwork performance information, such as real-time and total bandwidthusage, packet, jitter, and latency, may be identified as illustratedrelative to 17 b 14. Alternatively, an identifier representing fullaccess to all available network performance information may beidentified as illustrated relative to 17 b 16. Alternatively, NPIidentifier 17 b 6 may be utilized to differentiate between categories ofpackets that are communicated across a network. For example, identifiersmay be utilized to differentiate access between overall networkperformance information and more targeted network performanceinformation, such as network performance information associated withreal-time data packets. Any combination of the foregoing may also beutilized. For example, an identifier may be provided that allows fullaccess to real-time network performance information.

Although only one example of a portion of a permission table isillustrated in FIG. 17B, any combination of entity identifiers 17 b 2associated with individual network participants or groups of networkparticipants may be used in a table or other data structure with segmentidentifiers 17 b 4 of individual network segments, categories of networksegments, entire networks, particular connection paths, particular linestate information, or any categorization or grouping of the foregoing,and may be further used with NPI identifier 17 b 6 that are associatedwith individual categories of network performance information, datatypes, and other NPI identifiers offering full access or no access atall may be utilized.

The network performance tables may include network performanceinformation on many different levels. For example, the networkperformance information may be collected and status values indicative ofthe operational status or performance of the node segments may begenerated in addition to storing specific network performanceinformation (e.g., packet loss, bit rate, bandwidth, etc.) in thenetwork performance information tables. Rules may be established toenable certain users, partners, affiliates, or otherwise, to have accessto certain levels of data. The levels of data may be specified in thenetwork performance information table and define parameters that eachlevel may access.

Additionally, network performance information being communicated via PIPpackets may also be protected from different entities, nodes, orotherwise, from having access to certain information. In one embodiment,the network performance information may be encoded or otherwiseidentified so that the level of the information is specified and,thereby, restricted to be accessed by parties or equipment that do nothave permission to access nodes above a certain level. The levels mayrange from 1 to 10, for example.

PIP Data Packet Stitching

FIG. 18 is a block diagram of exemplary multi-node packet networks 1800a and 1800 b (collectively 1800) used to communicate data packets 1802including PIP packets to convey network performance informationgenerated by each node or network element 1804 a-1804 n (collectively1804) in a transmission path. As shown, there are two packet networks1800 a and 1800 b formed of multiple network elements or networkcommunications devices 1804 that may form a network of one or moreservice providers. Because there are two networks 1800,network-to-network interface devices 1806 a and 1806 b (collectively1806) are configured to communicate with one another, thereby forming abridge 1808 across the packet networks 1800 a and 1800 b. This bridgecould include direct connections of the same technology as in 1800 a and1800 b, or as another technology existing in a different type ofnetwork, such as SONET bridging two Ethernet networks, for example.

Each of Network A and B may have its own respective PIP packetsassociated with each of their respective networks and respectivemodified Y.1731 protocol stacks or other measurement processesmonitoring communications between nodes 1804 and 1806. A third set ofPIP packets and associated Y.1731 protocol stacks may monitor thecommunications between the network-to-network interface devices 1806.Real-time data packet performance and total network performanceinformation may be generated and communicated in the PIP data packets.In one embodiment, a first-end point in the network communications pathgenerates a special PIP packet that routes through devices 1804 andtriggering each modified Y.1731 measurement engine to inject its storednetwork performance information into the PIP packet by appending networkperformance information and a segment identifier into the PIP packets asit is passed downstream. This PIP packet with the appended networkperformance information then continues and triggers the same performancemeasurements (PM) or network performance information pull at the nextnetwork node and associated modified Y.1731 protocol stack inserts itsPM information concatenated behind the first nodes PM information. Thisprocess continues for each network node such that the network node'snode segment identifier and network performance information isconcatenated into the next PIP packet sent in the downstream pathdirection. Such PIP packet travels in turn to the subsequent networknode and associated Y.1731 protocol stack, where additional networkperformance information is appended and a node segment identifier isadded and then the combined network performance information istransmitted again via a PIP packet until the PIP packet reaches the farend network node and Y.1731 protocol stack. Each Y.1731 protocol stackalong the path can read the appended segment PM information and storethe network performance information or optionally choose to terminatethe PIP segment PM information by removing some or all of the appendeddata so the next PIP packet sent downstream contains only that segment'sinformation, or simply add its own network performance information intothe PIP packet. If a network node or associated modified Y.1731 stackprotocol terminates the PIP packet, the network performance informationis stored, and a new PIP packet that does not include the stored networkperformance information is communicated to start stitching new networkperformance information from network nodes and associated modifiedY.1731 protocol stacks located downstream in a network. It should beunderstood that data packets other than PIP data packets and stacks orprocesses other than the modified Y.1731 protocol stack may be utilizedto communicate the network performance information between the networkcommunications devices. The network communications devices 1804 may beconfigured to communicate the network performance information in the PIPdata packets on a regular basis (e.g., every second, minute, hour,100^(th) PIP data packet), based on an event (e.g., performanceinformation parameter crossing a threshold value), in response toreceiving a PIP data packet with network performance informationcontained in the payload portion, or in response to a request orcommand. The PIP data packets 1802 may be 802.1AG data packets and becommunicated over OSI Layer 2.

To correlate the performance manager measurements on a transmissionpath, the network performance information measured may use multiple binsto store data collected over a period of time. Multiple bins canconcurrently exist for different time window lengths, such as 5 minutes,10, minutes or otherwise. For example, special “bin roll” PIP packets orsequentially number packets may indicate which timeframe bin theinformation should be stored in. For example, PIP sequence or binpackets may be generated at time periods that include 1, 5, and 15minutes. The modified Y.1731 performance measurement function can havemultiple bins that correlate to the PIP sequencing or flags to ensurethe PIP packet data is stored in the correct bin. Also, longer timeperiods may be added to the modified Y.1731 stack, including 1 hour and24 hours. The modified Y.1731 stack or counters may be accessed toderive or compute the network performance information. Access may beperformed by using time period numbering.

To provide isolated segment performance information, each networkcommunications device 1804 may be configured to receive a PIP datapacket 1802 containing network performance information from othernetwork communications devices and append network performanceinformation generated at the respective network communication devicewith the other network performance information within the PIP datapacket 1802. This concatenation process may be considered “stitching” ofnetwork performance information along a transmission path. This functionhelps identify path verses shaping functions in the transmission path,and also provides the ability to retrieve network performanceinformation in-band verses using multiple external EMS systems toretrieve the information for fault isolation. To ensure that errors inthe bins do not occur, the PIP data packets 1802 may be marked (e.g.,<15minupdate>) so that each of the network communications devices alonga communications path appends information contained in the Y.1731 binthat is storing network performance information associated with themarking (i.e., <15minupdate>).

FIG. 19 is an illustration of an exemplary series of networkcommunications devices 1902 a-1902 n (collectively 1902) that areconfigured to append network performance information in PIP data packets1904 a-1904 n (collectively 1904). Each of network communicationsdevices 1902 may be configured to append most recent and/or historicalnetwork performance information to network performance informationreceived in the PIP data packets 1904 from other network communicationsdevices 1902, as indicated by payload portion 1906 a-1906 n of the PIPdata packets 1904 increasing in length after each of the networkcommunications devices 1902. Alternatively, such network performanceinformation may be otherwise inserted into or represented in PIP datapackets 1904. Header portion 1908 a-1908 n of the PIP data packets 1904may be configured normally. In practice, the network performanceinformation may be written into the PIP data packets 1904 using XMLlanguage or other language (e.g., <param start tag>param value</paramstop tag>). The principles of the present invention provides support forboth tag delimited and fixed width fields within the performance packet.For example, the following descriptor may include network performanceinformation generated over a time period.

<NSEG>A204 </NSEG><JITR>.002 </JITR><DEL>.04 </DEL><PL>125 </PL>... <RTBW>1.73 </RTBW><TBW>3.74</TBW><TIME>07.43.47.14.07.07 </TIME><NSEG>A205 </NSEG>...

Included at the start of the network performance information is anidentifier of the node segment “A204.” Additionally, an identifier of acarrier may be added to the PIP data packet, such a carrier name orcode. Jitter, delay, packet loss, real-time bandwidth, total bandwidth,and time at which the network performance information was generated mayalso be included in the network performance information. This networkperformance information may be compared with historical networkperformance information previously sent from node segment A204 todetermine whether a problem has developed over time. As shown, networkperformance information of network segment “A205” is appended to thenetwork performance information “A204.” It should be understood thatother embodiments for communicating the network performance informationmay be utilized.

Continuing with FIG. 18, so that the network performance informationassociated with different node segments can be easily identified, anidentifier that describes each node segment may be included in the PIPdata packet 1802 by positioning the identifier in front of the networkperformance information generated by each of the network communicationsdevice 1804. Continuously concatenating the network performanceinformation and communicating the appended network performanceinformation in PIP data packets across a transmission path fromend-to-end provides for a complete description of a transmission pathwith detailed viewing of network performance information associated witheach connection, network element, media, or other network segmentincluded in such transmission path. Concatenating network performanceinformation in PIP data packets may also be performed on abbreviatedtransmission paths to further help isolate a transmission problem. If,for example, a transmission path is having transmission qualityproblems, an analysis of the network performance information collectedalong each node segment of the transmission path may be performed toidentify the node segment(s) that are contributing to the transmissionquality problem. For example, the stitching process may be performed onMEPs, MIPs, NNIs, CPEs and be performed routinely or in response to acommand issued by a CCM. The last node that receives the networkperformance information may be configured to perform analysis on thenetwork performance information or communicate the information to a CCM,NOC, EMS system, correlation engine, or other network device. If thenetwork performance information is communicated over multiple networks,a CCM that manages the last node may communicate the compiled networkperformance information to the originating CCM for transmissionperformance analysis. The network performance information may be storedin tables at each of the nodes in the transmission path and/or one ormore network devices may receive and store the network performanceinformation in table(s). It should be understood that some performanceinformation collected from different types of network nodes, such asthose within the NNI, may contain performance information that is unlikeperformance information captured from other nodes such as MIPs or MEPs.The embodiments of this invention allow disparate types of performanceinformation to be concatenated into a single performance flow.

The network communications devices may include performance managers(PMs) that perform the function of managing the modified Y.1731 stack.The performance manager or other software module may perform thefunctions of accessing the stack to collect network performanceinformation, optionally at particular time periods, concatenating thenetwork performance information into the PIP data packets, andcommunicating the data packets in-band or out-of-band. The performancemanagers of the network communications devices may become tools thatrepresent correlated performance manager counter usage in-band in shorttime intervals, which results in (i) eliminating in-flight measuringaccuracy issues, (ii) eliminating multi-carrier segment troubleshooting,(iii) optionally enabling in-band performance managers for access versusout-of-band enhanced messaging services or graphical user interfaces,(iv) and correcting the issue of stacking access technologies thatintroduce multiple in-line PIP packet flows that have to be polled.

FIG. 20 is an exemplary process 2000 for communicating networkperformance information of a node segment of a packet network. At step2002, network performance information indicative of transmissioncharacteristics of a node segment on a packet network may be generated.The network performance information may be included in a data packet atstep 2004 and communicated using in-band signaling to a networkcommunications device at step 2006. The network performance informationmay be appended to other network performance information received in adata packet from another network communications device and communicatedin turn via a third data packet to another network communicationsdevice. The data packets may be PIP data packets.

FIG. 21 is an exemplary process 2100 for identifying communicationproblems within one or more packet networks. At step 2102, networkperformance information may be generated at network communicationsdevices in communication with one or more packet networks. At a firstnetwork communications device, a first data packet including firstnetwork performance information generated at the first networkcommunications device may be generated at step 2104. At step 2106, thefirst data packet including the first network performance informationfrom the first network communications device may be communicated to asecond network communications device. At the second networkcommunications device, a second data packet including the first networkperformance information received from the first network communicationsdevice and second network performance information generated at thesecond network communications device may be generated at step 2108. Thesecond data packet including the first and second network performanceinformation may be communicated from the second network communicationsdevice to a third network communications device at step 2110. Thisprocess of generating, concatenating, and communicating networkperformance information may start at a first end of a transmission pathand finish at the second end of the transmission path so that eachnetwork communications device has provided network performanceinformation that may be used to determine where a transmissionperformance problem exists along the transmission path. For example, ifbandwidth for real-time applications is being lost at a node segment, aservice provider may identify which node segment along thecommunications path is losing bandwidth for real-time applications.

FIG. 22 is an illustration of an exemplary packet network 2200 with oneservice provider and two operators. The packet network 2200 includesoperator network equipment 2202 a-2202 f (collectively 2202) andsubscriber equipment 2204 a and 2204 b (collectively 2204). The MetroEthernet forum has defined Operations, Administration and Maintenance(OAM) Maintenance Entities (MEs) as shown. More specifically, the MetroEthernet has defined multiple administrative domains, such as SubscriberMaintenance Entity (ME) 2206, Test Maintenance Entity, User NetworkInterface (UNI) 2208 a and 2208 b (collectively 2208), OperatorMaintenance Entity 2210 a and 2210 b (collectively 2210), andNetwork-to-Network Interface Maintenance Entity (E-NNI ME) 2212. Eachoperator is provided with visibility across its respective network viathe operator MEs 2210, but cannot view information in the Operator ME ofother operators unless the other operators provide the properpermissions to allow this view.

In accordance with the principles of the present invention, an OAMdomain, shown as a Stitched ME 2214, extends between the subscriberequipment 2204 through the transmission path of the operator equipment2202 of both operators. The Stitched ME 2214 provides for communicationof network performance information generated at each element ofsubscriber and network equipment in the packet network 2200, and PIPdata packets (not shown) may be generated and communicated from thesubscriber equipment 2204 a and successively through the operatorequipment 2202 on the packet network 2200 as a single flow that isstitched or concatenated together via higher order packets to thesubscriber equipment 2204 b (upstream to downstream). It should beunderstood that two flows may be operating in opposite directions sincethe full duplex nature of some communication technologies allowdivergent receive and transmit paths. Each MEG Intermediate Point (MIP)(set of stitched MEG End Points (MEPs)) may transparently append orblock and re-start the PIP data packet flow, or selectively includingnetwork performance information. Bins having predetermined time periods(e.g., 5 minutes, 15 minutes, 1 hour, 24 hours) may be created to createa “stitched” PIP data packet that pulls network performance informationfrom each of the communications devices (subscriber equipment 2204 andoperator equipment 2202) on the packet network 2200 in a correlatedmanner. Sequences and counter resets may additionally be adopted forproviding the PIP data packet stitching. To perform the stitchingoperations, the MIPs may pull the modified Y.1731 information from theupstream node segment and append it to the stitched packet travelingdownstream. As previously described, node identifiers and/or carriernames or codes may be included in the PIP data packets to identify thecarrier and node segment that the modified Y.1731 performance managerdata was inserted. In one embodiment, MIPs, or certain MIPs, may remain“unstitched” and operate as a pass-through.

Although the Stitched ME domain extends from end-user to end-user, theprinciples of the present invention may provide the ability foroperators to be limited to accessing information from their own networkor limited information from other service provider networks.Subscribers, similarly, may be limited to having access to their ownequipment or a summary of operator information. There may be a number ofdifferent techniques used to provide such limited visibility foroperators and subscribers, including safeguards built into performancemanagers at each network communications device.

FIGS. 23A and 23B are illustrations of a multi-carrier network 2300having multiple Ethernet service providers (ESPs) 2302 a, 2302 b, and2302 c (collectively 2302) and a multi-point network 2304 having amulti-point device 2306 in communication with network interface devices2308 a-2308 d (collectively 2308). Using a stitched PIP packet streamenables an end-user to determine performance of each node segment todetermine if equipment operated by one of the service providers 2302 ishaving a communications problem. In the case of multi-pointcommunications, a transport performance manager may be isolated from theswitching performance manager. It should be noted that non-Ethernetperformance information may exist and be included in the PIP packetstream.

Call Control Manager Functionality Enhancement

To provide a better experience for end-users, a mechanism is beingintroduced to provide near real-time monitoring capabilities of the pathand link status of the underlying packet network upon which a voice-onlyor multi-media call is carried. This information can be sent to the CCMand acted upon to choose or alter the call characteristics and routingof calls, such as change codec use, provide call treatment and routing,and alter overall use of the call path, etc., thereby providing a betterquality of service for the end users.

One technique for providing this network monitoring capability is theuse of a link state reporting structure in the form of PIP packets. Bothline state (i.e., transmission path to a user), and trunk state (i.e.,shared transmission path state between network nodes) can be provided tothe CCM to convey the transmission path state of the packet network. ThePIP packets themselves provide a line or trunk state, respectively, ateach end of a line or trunk transmission path. To enable CCM managementcapabilities, line and trunk states may be communicated to the CCM viacall control protocols or some other type of packet network signaling.The PIP and PM measurements protocols provide the means to monitor thequality of the link states and report findings to a separate networkelement. As previously described, included in this near real-time reportmay be real-time bandwidth usage, packet loss, latency, and jitter orany other network performance information. The monitoring of thisinformation can happen within any area of a network and can provide ameans to report the lower layer status of the network. As shown in FIG.15, there are many places that these measurements can be taken. The PIPpackets provide the information used to determine path capabilities fromthe network end points. Bearer path monitoring may be accomplishedbetween the following elements of FIG. 15:

-   -   End User 1512 and Network Access Node 1514, or optionally to        router 1506    -   End User 1518 and Network Access Node 1514, or optionally to        router 1506    -   Network Access Node 1514 and the Network Router 1506    -   Network Router 1506 and Network-Network Interface 1510, or        optionally to a Media Gateway deployed in another carrier's        network    -   Network Router 1506 and Media Gateway 1520    -   Network Router 1506 and Media Gateway 1516

PIP packets may also provide information between two end devices eventhough a network element is located between the two end devices. Thatis, if the provider would want to see the overall “health” of the pathbetween media gateway 1520 and media gateway 1516, the PIP packets canbe configured to monitor this route even though the router 1506 is partof the routing of this path. Once collected, the raw information fromthese paths can be configured to show the overall health of the route.Information contained within the PIP packets may be used to determinemetrics, such as real-time bandwidth usage, jitter, packet loss andoverall delay, of the path being measured. These calculations may beperformed at the individual element, or information may be transportedto another collection device to be used by other call processingfunctions as shown in the CCM 1502. These real-time events may be usedto provide input into the decision functions used in call routing withinthe CCM. The measure of real-time bandwidth being provided by the PIPpackets also enables a summation of the real-time bandwidth on thatpath. Historically, this metric is part of a TDM CCM function, but wasnot replicable or available without both the number and amount ofreal-time bandwidth usage on a node segment. These combined functionsprovide for such a measure, thereby the CCM may contain a table of theamount of “Erlangs” being used on trunking facilities. Other timeintervals may be used to accommodate other non-Erlang-like measures.

In accordance with the principles of the present invention, new stepsare added to the call processing 1504 of the CCM 1502, over and abovethe normal call processing currently done. Since the CCM now has thecapability to track the lower layer performance and bandwidthavailability of the underlying network a new type of status table may beadded to call processing that systematically updates during specifictime intervals throughout the day. These updates, which may occurperiodically (e.g., once per second) may be placed in one or more tables(see TABLES II-IV) in a form that may show utilization, latency, jitter,and packet loss. While there are other types of information that couldbe shown, such as Mean Opinion Score (MOS) voice values, for simplicity,these three basic parameters are illustrated and discussed herein.

In normal operation, when an end-point, either a trunk or line,initiates a call, call processing operating in the CCM 1502 determinesthe terminating end-point to complete the call. In conventional callingscenarios, the call would then be set-up and the call path establishedfor the end-users to converse. This is conventional call processingbased on provisioned information that would give call processing theability to route the call. The principles of the present invention takeadvantage of collecting the network performance and utilizationinformation from the node segments to aid in performing call processing.Call processing may perform normal information lookup to determine theoriginating and terminating end-points of the call, but before routingthe call, the end-point or node segment information from theseend-points may be retrieved and call processing may query a networksegment status table (e.g. TABLE IV) to determine the line or trunkstate availability of the node segments that could be used to connectthe originating and terminating end-points for the call. Depending onthe availability of the paths used on the call, special call handling,load balancing, call spacing, or other special call handling can beinvoked to sustain call processing, and provide relief for the call pathor in extreme congestion, alternate routing could take place to providefor a satisfactory voice path on the packet network. This management canbe done at call set-up and/or anytime during the call. As transmissionstate is available at both the CCM and call protocol stack at the userlocation, multiple enhanced call functions may be possible. For example,outgoing user calls could automatically query the line state on the CPEto provide the user with graphical or text based feedback as to calloptions for multi-media setup given they have a specific transmissionquality to utilize. For example, a multi-media call could revert to afrozen image and voice-only call until congestion clears. The samecondition may enable the CCM to know the state before setting up a calland either make a decision by itself or query the user or user'sequipment about how to alter ongoing sessions to allow morecommunications. The line state information availability to the switchand user may be used to provide session control feedback. The sameinformation of threshold crossings may be used to convey that a call maybe dropped prior to the incident occurring. These functions may havesignificant value to the customer experience. The trunk state of sharedresources is paramount for inter and intra-switch path state knowledge.Packet networks may be considered to operate autonomously given that thebandwidth being used by the CCM is also being used by other serviceswithout knowledge by the CCM. To operate appropriately, the CCM may usetransmission state feedback so it can be pre-cognitive of thecommunications path state during call handling. Without the trunk-stateinformation between two switches, each switch operates under theassumption that enough bandwidth exists to sustain all calls. Often,neither switch will “own” the bandwidth flow control mechanisms forflows between the switches, so this assumption is dangerous in terms ofproviding carrier grade call handling. Conditions can arise in whichinadequate bandwidth or device resources are available to support allcalls and packets are dropped. If the switch knows the path state (lineor trunk), call handling alternatives may provide customers withfeedback that was previously unavailable and provide better call qualityand call handling. It should be understood that the CCM may use bothline and trunk state tables and make call handling and customer callfeedback decisions based on the severity of node segment congestion,including, for example: (i) CODEC modification, (ii) rerouting the call,and (iii) congestion control.

CODEC Modification

In a line-to-line call between two end-users 1512 and 1518 (FIG. 15),alternate routing to the line end-points is not a viable alternativesince each communicates via the network access node 1514. Since mostend-lines have one path for transport to the packet network, othermodifications are performed to provide better call quality. Onemodification that provides a better call capability would be a CODECchange to raise or lower bandwidth of the CODEC (i.e., a CODEC thatoperates at a different speed). In one embodiment, the bandwidth israised or lowered by sending a command to the CODEC to raise or lowerits bandwidth. Alternatively, a different CODEC may be employed forperforming the call. This replacement could occur in mid-call.

FIGS. 24A-24C are flow charts of an exemplary process for performingline-to-line call flow. The process 2400 a starts at step 2402, wherethe call processor is idle. It should be understood that the callprocessor may be hardware, software, or a combination thereof. At step2404, an originating line or calling party goes off-hook and dials anumber of a called party or destination line. This call, in turn, isreceived by the CCM as an incoming call. Information for this call ispassed from the end-unit to the CCM. In IP telephony, the signalingprotocol may be Session Initiation Protocol (SIP), but other signalingprotocols, such as Media Gateway Control Protocol (MGCP) or Megaco(H.248) may be used.

At step 2406, a decision is made if the call is allowed by determiningwhether the calling party is registered, authorized or otherwise. TheCCM, more specifically, retrieves terminating end-point addressing andlocation based on conventional table lookups within the CCM. At step2408, a decision is made as to whether the call is allowed. If it isdetermined that a call is not allowed, then at step 2410, a “reject”message may be sent to the originating line and the process ends at step2412. If at step 2408, a determination is made that the call is allowed,then at step 2414, routing translations from call control is checked. Atstep 2416, routing information is found and termination line informationis received. In addition, node segment assignment information for theoriginating and terminating lines is retrieved.

At step 2418, a determination as to whether the terminating line isavailable is made. If the terminating line is not available, then atstep 2420, a “reject” message may be sent to the origination line andthe process ends at step 2422. If, however, at step 2418 the terminationline is determined to be available, then at step 2424, the node segmentstatus table is accessed to locate usage status of node segments to beused for connecting a call between the origination and terminationlines.

At step 2426, node segment state information stored in the node segmentstatus table is received. The information from the node segment statustable includes origination line information and termination line stateinformation. The node segment status information is used to determine ifa transmission path state to be used for the call has any congestion.Depending on numerical or other indicia status retrieved by the callprocessing, a determination of the congestion of the transmission pathwill be (i) normal or (ii) impaired, or (iii) congested, for example.The determination of the transmission path being normal, impaired, orcongested is made based on network performance information having valuesdetermined to be within ranges, where the range may be a single value(e.g., 1, “A,” “normal”). It should be understood that the range may bedefined by a single value, such as “Congestion” representing statusvalues between 7 and 10, for example. As previously described, thevalues may be processed to be within a scale, such as 1-10, indicativeof the status of transmission performance.

At step 2428, results from the node segment status table for theoriginating line is received. At step 2430, a determination of a largestnode segment status is made. Determination of the largest node segmentstatus is made by determining a highest value of status indicators inthe node status segment table associated with the originating lineinformation. Determining the largest node segment status is performed toidentify a limiting transmission parameter (e.g., bandwidth usage,packet loss). As previously described with regard to TABLES V and VI,the larger the value, the worse the network performance informationassociated with a node segment, thereby resulting in poor voice qualityduring a call, this information may be provided back to the caller assystem feedback. Note, that the largest value may be a high qualityvalue and is indicative of a well performing network; i.e., all pathsare equal and capable of supporting high quality calls. At step 2432, adetermination as to the status of the originating line node segment ismade, which, in one embodiment produces one of three results, normal,impaired, or congested. If normal, the process continues at step 2434 inFIG. 24B. Alternatively, if the status of the originating line nodesegment is impaired, the process continues at step 2436 in FIG. 24C.Still yet, if the status of the original line node segment is congested,then the process continues at step 2438 in FIG. 24C Continuing with anormal status of the originating line node segment, at step 2440 in FIG.24B, a call invite set-up message with normal request is sent to theterminating line. Normal call control is continued at step 2442. Theprocess ends at step 2444.

If it is determined at step 2432 that there is an impaired condition,then call processing being performed by the CCM may change the messagethat would be sent out to the terminating end-point to request a CODECto use a lower bandwidth for the call path. This lower bandwidth requestmay be performed in concert with a user interface or performed via theuser CODECs without user participation. This is shown at step 2452,where CODEC capabilities of the originating and terminating lines arechecked and a determination is made at step 2454 as to whether a lowerspeed CODEC is available. If it is determined that no lower speed CODECis available, then at step 2456, a “reject” message may be sent to theoriginating line. The process ends at step 2458. If, however, adetermination is made at step 2454 that a lower speed CODEC isavailable, then at step 2460, an invite with a lower speed CODEC may besent to the terminating line. At step 2462, a wait may be performed fora subsequent message from the terminating line. At step 2464, a positiveresponse message may be received from the terminating line. At step2466, a message is sent with new CODEC information to the originatingline and, at step 2468, normal call control is performed. The processends at step 2470. Not shown in this embodiment is that other call setmeasures could be considered in a serial or parallel fashion by the CCMin addition to CODEC negotiation to establish a quality call.

In summary, FIG. 24C operates to change a message that is sent out tothe terminating end-point to request a lower bandwidth for the callpath. For example, if the originating calling party requested use of aG.711 voice CODEC that uses 64 Kb/s for the bandwidth, the callprocessing may change the request to the terminating called party to aG.729 CODEC that uses only 8 Kb/s. While the voice quality may not be asgood as the higher bandwidth CODEC originally requested, the bandwidthselected may be reduced enough to allow the call to be completed withbetter voice quality than if it was impaired while using the originallyrequested higher bandwidth CODEC. It should be understood that the linestate information may be used to facilitate customer call setupfeedback, and possibly call setup control with CODEC selection choice.It is understood that line state can apply to wireless network devicesconnected behind multiple access technologies, where a line state PIPpacket may originate at the end-user device and terminate at a specificaccess node or at some point between the CCM dedicated switch or router.The access node may enable line state transmission path utilization andperformance management tracking. Also, it should be understood thatcalls could be of any type, including voice, multi-media, or otherwise,where timely and quality delivery may be call path considerations.

To provide the originating caller with a CODEC change, call processingmay wait for the return information from the called party to bereceived. Once the information is received from the called party, thenthe call processing may alter the message to include the change to alower bandwidth CODEC and pass it on to the originating party. From thispoint on, normal call processing would continue and the call would beset-up with the lower bandwidth CODEC.

If at step 2432, a determination is made that the originating line nodesegment is congested, then the process continues at step 2438 in FIG.24C where a “reject” message may be sent to the originating line at step2456 and the process ends at step 2458. The “reject” message is sentbecause call processing determined that the call could not continue eventhough a lower bandwidth CODEC could be used. A user notification, suchas an audible or visual ‘Network Busy’ message, may be sent to thecalling party. Depending on the severity of the deterioration of thetransmission path, the CCM may send out a response to the calling partyrequest not to allow the call to continue. This “throttling” of callscoming into the packet network provides established calls more bandwidthto use, and the calling party of the rejected call may receive a busysignal. The calling party may place the call at a later time and adetermination may be made at that time as to whether the status of nodesegments associated with the calling party is normal (i.e., status valuewithin a range). In an alternative embodiment, the CCM may automaticallycontinue to regularly attempt to set-up the call. When the congestionclears, the CCM notifies the calling party that a call can now be set-upand completes the call per the calling party instructions.

Continuing with FIG. 24A, steps 2446, 2448 and 2450 are steps performedin response to receiving terminating line information and mirror thesteps 2428, 2430, and 2432, respectively. In other words, the process2400 a makes the same or similar determinations on both the originatingline and terminating line to ensure that status of node segmentsassociated with each of the calling and called parties is operatingproperly.

Best Path Metrics

In determining transmission paths through a packet network, a CCM orother node may make a determination of the transmission path for a callor other communication to be made over the packet network based oncurrent, historical usage, or network performance of node segments onthe packet network. In one embodiment, a transmission path to route thecall or communication may be determined by using network performanceinformation or information derived therefrom (e.g., network segmentstatus information) available in a table or at each node along apotential transmission path. In one embodiment, a calculation may bemade to determine metrics along one or more transmission paths throughthe packet network to determine that the metrics result in a cumulativevalue below a threshold or the best metric of the potential transmissionpaths. Currently, most best-path algorithms use total utilization andbandwidth size for determining the quality of the path. In accordancewith the principles of the present invention, characterization ofreal-time jitter and delay performance characteristics may be used todetermine best path metrics. Modification of the best path metrics toinclude the real-time usage, and performance enables enhanced loadbalancing and path choice decisions for real-time flows. In oneembodiment, these real-time network performance informationcharacteristics may have a higher priority on the network. This modifiedmetric enables the network to make enhanced routing decisions fortraffic routing that was not possible without the transmission state ornetwork performance information. One example of best route calculationsimprovement may include averaging, and, optionally, burstinesscharacterization. Best path calculation methods may includecalculations, such as root-sum-square (RSS) and weighted vectorcalculations, that may be utilized to determine the path or paths withthe optimum best path metrics. Further, a weighted average of thenetwork performance information or status levels may be determined. Inone embodiment, the best path metrics may create a real-time utilizationstate by which engines, EMS systems, and other network protocols mayretrieve and utilize to gain system feedback as to the nature of thereal-time network state. Also, a search for a transmission path havinglowest sum of status levels may be used to determine best path metrics.In response to determining the transmission path with the best metrics,that transmission path may be used for establishing the transmissionpath for a call or communication.

Rerouting Calls Using Network Status Segment Table

While calls between two end-users, such as end-users 1512 and 1518 (FIG.15), on the same network access node 1514 does not allow rerouting ofcalls between the two end-users, calls from an end-user 1512 through amedia gateway 1516 or 1520 or other network interface device may provideadditional options for alternate call routing destinations versesaltering a CODEC selection. For a call between an end-user and a mediagateway or other trunking device, there may be more than one route ortermination point to successfully complete the call. These routingoptions are often the case with PSTN switching, where an end-officeswitch (class 5) may have an alternate tandem call termination point toreach that same end-office. That is, if the call is routed to a specificgateway and that route is congested, it may be possible to locateanother media gateway with a route to the destination. By determining atransmission path and using the node segment status table (e.g., TABLESV and VI) on the route to the terminating media gateway or trunk, callprocessing could be instructed to determine whether a secondary trunkcapability is available for the call and determine if the secondarytrunk has an uncongested path to the destination of the call. This samefunction enables geographical fail-over or call routing when networkcongestion or network failure significantly impairs the packettransmission path to a remotely deployed media gateway. In addition,predictive algorithms that trend performance information may recognizethat a link is failing and systematically re-route traffic to an optimumlink while managing the quantity and quality of the calls.

In a typical line-to-trunk call, the combination of line segmentcongestion and trunk segment congestion may be taken into account. Itshould be understood that a network switch may track all transmissionpaths to a central point, trunking point to trunking point, hybrid ofline to central point, or line to trunk in a transmission state table.Since the end-user initiates the call, the first half of the call woulduse node segment analysis described previously to determine if thetransmission path at the calling node segment is operating properly orhas impairment. If the calling node segment is found to be impaired,then call processing may determine that a lower bandwidth CODEC may beutilized to improve the call quality or take other steps, such as allowthe call to be made as a voice-only call rather than a multi-media call.If the originating node segment is congested, then the call processingmay reject the call since there is no other path for the end-user touse. However, if there is a transmission quality or utilization problemat the terminating trunk node segment, then a rerouting option for thecall may be available. In one embodiment, utilization means real-timeutilization as compared to total bandwidth utilization with packet lossor the statically provisioned bandwidth allotted in that physical orvirtual channel. Any indicators can serve to calculate the state of theuser's transmission “line” or shared resource “trunk” transmission path.As stated, the CCM can now have a secondary “state” for that segment,line, or trunk by which it predetermines how call processing for thatend-point should be handled. This secondary state is indicated in tableVII below. TABLE VII includes an exemplary list of scenarios for thecall processing to follow based on the combined status of theoriginating line and the terminating trunk.

TABLE VII NETWORK STATUS AND RE-ROUTING CALL OPTIONS ScenarioOriginating Line Terminating Trunk Call Processing 1 Normal NormalNormal 2 Impaired Normal Adjust CODEC or Reroute 3 Congested NormalReject Call 4 Normal or Impaired Impaired Adjust CODEC or Reroute 5Normal or Impaired Congestion RerouteScenario 1

In this scenario, normal call processing may be used since none of thetransmission paths are constrained or impaired. The call may be routedwithout any changes to the voice coding of the call through thetransmission path.

Scenario 2

Since the originating line is impaired, call processing may adjust rateof a CODEC for the call. The rate adjustment may be performed bylowering the rate of a CODEC or routing the call to another CODEC havinga lower rate. Call processing may check the segments of the outgoingtrunk to determine if the media gateway on the transmission path hascapability to alter the CODEC used to convert the packet information(e.g., IP Packet Information) to a TDM format. If CODEC alteration ispossible, then the CCM may negotiate the CODEC speed between theoriginating call device and the terminating trunk and establish the callvia the CODEC having the lower speed. If the media gateway does not havemultiple CODEC speed capability, then the call controller may have theoption of routing the call via another trunk group if an alternate routeto the terminating call device is available. If another route exists,then the call processor may reroute the call to the next trunk group andthe node segment status check may be performed prior to establishing thecall via the trunk group. If the trunk group has CODEC modificationcapabilities, then the call may be established via a CODEC with a lowerspeed and the call may be established. If another trunk cannot be foundwith CODEC speed alternatives, then the call may be dropped.

Scenario 3

If the originating line is determined to be congested, then since thereare no alternative routes for the originating part of the call, then acall “reject” may be sent to the user and the call dropped.

Scenario 4

If the terminating side of the call is determined to be impaired, then adetermination as to a lower bandwidth CODEC may be used. If theterminating trunk group has the capability to use a different CODEC,then a determination as to the CODEC capabilities of the originatingline may be performed. If a lower bandwidth CODEC is available, the callmay be established with these CODECs and the call may proceed normally.If there are no CODECs available at the originating side with a lowerbandwidth, then the call processing may perform a reroute as describedin Scenario 2.

Scenario 5

If the terminating trunk is determined to be congested, then callprocessing may search for a reroute for the call over a terminatingtrunk that is not congested. The call processing may include locating atrunk group having a normal or lower speed CODEC for establishing thecall.

FIGS. 25A-25C (collectively FIG. 25) are flow diagrams of an exemplaryprocess for providing call processing for rerouting a call between anoriginating line and terminating trunk. The process in FIG. 25 may beperformed by a call processor at the CCM 1502 (FIG. 15) or, optionally,other call managers if distributed on the packet network. The process2500 a starts at step 2502. At step 2504, an incoming call is received.In one embodiment, the incoming call is an SIP invite. At step 2506, adetermination may be made if the call is allowed by determining if thecaller is registered with the service provider. A determination may bemade at step 2508 to determine if the call is allowed. If not, theprocess continues at step 2510, where a “reject” message is sent to theoriginating call device and the process ends at step 2512. If the callis determined to be allowed at step 2508, then the process continues atstep 2514, where routing translations from the call controller arechecked. Terminating trunk information is received at step 2516.Additionally, node segment assignment information for the originatingline and terminating trunk group may be received.

At step 2518, a determination may be made as to whether a terminatingtrunk in a transmission path between the originating call device andterminating call device is available within the trunk group. If so, thena node segment status table may be used to determine usage status of thenode segments along the transmission path. At step 2522, results fromthe node segment status for the originating line may be received. Atstep 2524, determination of the originating line segment status may bedetermined. In one embodiment, three node segment statuses may bedetermined, including “normal,” “impaired,” and “congested.” At step2526, a determination as to the status of the originating node segmentsmay be determined. If it is determined that status of the originatingnode segment is normal (i.e., status is within a range that provides fornormal, full transmission rate operation), then the process continues atstep 2528 in FIG. 25B. Otherwise, if it is determined that theoriginating node segment is impaired, the process continues at step2530. If, however, it is determined that the originating node segment iscongested, since there are no alternatives, the process continues atstep 2532, where a “reject” message is sent to the originating calldevice. The process ends at step 2534.

In one embodiment, the line and trunk state checking may become part ofthe call processing procedure. Typically, Call Admission Controlfunctions are blindly applied without regard to path state to reservebandwidth. In one embodiment, the line and trunk state utilization andperformance management may be provided as a state to the reservationengine in a switch to validate or accelerate the “CAC approval” versesstatically assigning the number of calls allowed. This modificationprovides enhanced value given that CAC function assumes a static CODECutilization and cannot predict the use of silent suppression or unknownreal-time use in the transmission path. It should be understood that theCAC function may be part of the CCM or reside outside the CCM on acentralized CAC resource, such as a RSVP or RAC server.

From step 2528 in FIG. 25B, the process continues at step 2536 where thenode segment status table is accessed to determine usage status of theterminating trunk. The results from the node segment status table forthe terminating trunk are returned at step 2538. A determination at step2540 is made of the node segment status of the terminating trunk. If thedetermination at step 2542 of the status of the terminating segment isnormal, then the process continues at step 2544, where an invite toestablish the call via the terminating trunk is performed. At step 2546,normal call control is performed and the process ends at step 2548.

If at step 2542, a determination is made that the status of theterminating segment is impaired, then at step 2550, a check of CODECcapabilities of the terminating trunk may be performed. If at step 2552it is determined that a CODEC having a lower rate is available, then atstep 2554, a check as to the CODEC capabilities of the originating lineis performed. At step 2556, if the determination is made that a CODEC isavailable with a lower rate at the originating line, then the processcontinues at step 2558 to send a set-up request for a lower speed CODECto the terminating trunk. At step 2560, the call processing waits for asubsequent message from the terminating trunk indicating that theterminating trunk has been able to set-up a CODEC with a lower speed atstep 2562. At step 2564, a message is sent to the originating line withnew CODEC information. At step 2566, the call control continues normallyand the process ends at step 2568. If at step 2556 a determination ismade that no CODECs are available at a lower rate for the originatingline, then a “reject” message is sent to the originating call device atstep 2570 and the process ends at step 2572.

If (i) at step 2542 a determination is made that the terminating segmentis congested or (ii) at step 2552 that no CODEC is available at theterminating trunk, then the process continues at step 2574, where callprocessing is checked to determine if there is another route availablefrom the originating call device to the terminating call device via adifferent trunk. At step 2576, results for a trunk group selection isreturned and a determination as to whether an alternative trunk group isavailable at step 2578. As understood in the art, a trunk group is twoor more trunks of the same type between two different nodes. If analternative trunk group is available at step 2578, then at step 2580, amessage may be sent to restart terminating trunk processing with a newtrunk group. The new terminating trunk information is received at step2582. Additionally, node segment assignment information for thealternative terminating trunk group may also be received. The processcontinues at step 2584, which repeats the process from step 2536 usingthe new terminating trunk for determining whether a call may beestablished via that trunk.

If at step 2578 it is determined that an alternative trunk group is notavailable, then at step 2586, a “reject” message may be sent to theoriginating call device. The process ends at step 2588.

Continuing from step 2526, if a determination that the status of theoriginating segment is impaired, then the process continues at step 2530(FIG. 25C). At step 2590, CODEC capabilities of the originating line arechecked. At step 2592, a determination is made as to whether a lowerspeed CODEC is available. The lower speed CODEC may be programmed to belower or be another CODEC that operates at a slower speed or change frommulti-media to voice-only or reduce to voice-only speed. If a lowerspeed CODEC is not available, then at step 2594, a “reject” message issent to the originating call device. The process ends at step 2596.

If it is determined that a lower speed CODEC is available at step 2592,then at step 2598, the network segment status table is accessed to findusage status of the terminating trunk. At step 25100, results from thenetwork segment status table for the terminating trunk are returned, anda determination as to the terminating trunk segment status is made atstep 25102. At step 25104, a determination is made as to whether theterminating trunk segment status is normal, impaired, or congested. Ifit is determined that the terminating trunk segment status is no moreimpaired, then at step 25106, CODEC capabilities of the terminatingtrunk are checked. At step 25108, a determination may be performed todetermine whether the CODEC is available. If a CODEC is available, thenat step 25110, a set-up request for a lower speed CODEC may be sent tothe terminating trunk. At step 25112, the call processing waits for asubsequent message from terminating trunk until the terminating trunknotifies the call processing that the lower speed CODEC is available andready at step 25114. At step 25116, the new CODEC information may besent to the originating line. The call control processing may continuenormally at step 25118 and the process ends at step 25120.

If at step 25104 a determination is made that the terminating segment iscongested or at step 25108 no CODEC is available at the terminatingtrunk, then the process continues at step 25122 to request from the callprocessing as to whether there is another route available via anothertrunk. At step 25124, results for another trunk group selection arereturned. At step 25126, a determination is made as to whether analternative trunk group is available. If not, then at step 25128, a“reject” message may be sent to the originating call device and theprocess ends at step 25130. If at step 25126 a determination is madethat an alternative trunk group is available, then at step 25132, amessage to restart the terminating trunk processing with a new trunkgroup is initiated. At step 25134, new terminating trunk groupinformation is received along with segment assignment information forthe alternative terminating trunk group. The process continues at step25136, which causes the process to use the new terminating trunk groupto determine whether a call may be established via that trunk group forthe call by the originating call device to the terminating call device.

Continuing at step 2518 of FIG. 25A, if it is determined that noterminating trunk is available within a trunk group, then at step 25138,the call processing checks to determine if another trunk route isavailable. At step 25140, the call processing returns informationindicative of the trunk availability. A determination at step 25142 ismade as to whether an alternative trunk group is available. If analternative trunk group is available, then at step 25144, the callprocessing may be restarted with the alternative trunk group and theprocess returns at step 25146 to step 2514 (FIG. 25A).

In summary, the process of FIG. 25 is used to determine status of atransmission path between an originating call device and a terminatingcall device via a trunk group. In determining the status, if the trunkgroup is having a communication problem as determined by a networksegment status table that derives its information from networkperformance information received from node segments on the packetnetwork, then the call processing determines whether it can lower thebandwidth of a CODEC or find an alternative route via another trunkgroup that has better communication performance for routing the call toa requested end-point.

Additional Call Rerouting

In one embodiment, the principles of the present invention provide fornetwork performance information to be utilized in rerouting calls tosubscribers in the event of a node segment being determined to beimpaired or congested, or otherwise unavailable for example. In such anevent, when a call comes into the CCM, the CCM may use a directory tolook up other potential contact's telephone numbers or addresses towhich the incoming call may be routed in an attempt to connect thecalling party with the called party. For example, if a calling party hasattempted to reach a called party on his or her mobile handset and theCCM determines that the transmission path to the subscriber's mobilehandset is not working properly, then the CCM may locate an alternativenumber of the called party, such as a home or work telephone number orother identifier, such as an SIP Universal Resource Identifier, androute the incoming call to the called party's alternative number oridentifier. In one embodiment, the CCM makes the decision as to whichnumber to call based on time of day or other factors (e.g., a subscriberpreference parameter).

In another embodiment, the CCM may receive a call to a subscriber thatthe CCM knows to be on a heavily congested or otherwise degraded nodesegment. The CCM may make a decision to place the call directly into acalled party's voicemail rather than tie up the heavily congested orotherwise impaired node segment with additional real-time contentcommunication. Alternatively, the CCM may notify the heavily congestedor otherwise degraded node segment to slow down, halt or otherwiseoffload non-real-time content communications being communicated throughthe node segments so that the telephone call, which is a real-timecontent communication, may be properly and timely placed to the calledparty.

Congestion Control

Calls from trunks, such as the network-to-network interface or sessionborder controller 1510 of FIG. 15, to lines over a packet networkpresent different challenges than line to line or line to trunk calls.Since the call control manager does not have complete control of apacket trunk path being selected for in-coming calls into thesoft-switch, congestion control is somewhat limited. If a call entersthe soft-switch from another network, trunk selection is actuallycontrolled by the other or far-end network. The call control manager,however, may have some level of call control utilizing the principles ofthe present invention.

Generally, when a call comes into the network, call processing operatingin the CCM receives an incoming call message with data identifying theport and address of the incoming call. Based on the port, address, andcalled number information, the CCM determines the transmission path,including the node segments, over which the call is assigned. Inaccordance with the principles of the present invention, the CCM mayexamine the status of the node segments associated with the transmissionpath. If the status of the node segment is classified as impaired, callprocessing may determine if the terminating line has the capability ofusing a lower bandwidth CODEC. If so, then the CCM may send a set-upmessage to the end-point requesting use of the lower bandwidth CODEC. Ina return response to the other network, call processing may pass the newrequest for the lower bandwidth CODEC. If accepted by the other network,the call may proceed. Otherwise, the call is terminated.

For other calls coming into the network via the trunk, the CCM mayfollow the same process of determining whether a CODEC having a lowerbandwidth is available. If a network segment status is congested, callprocessing may not try to process the incoming call and send a releaseto the other network via the originating trunk.

In one embodiment, the CCM may manually or automatically enact callthrottling procedures based on congestion of the originating trunksegment interconnecting to the other network. These throttlingprocedures may be in the form of automatic congestion control (ACC),selective incoming line control (SILC), call gapping, number or IPaddress blocking, or any other well-known throttling call controlmechanisms. Based on timers or incoming call counts, the CCM may allowcalls to be attempted at certain times to test the congestion of thepath. If the node segment becomes uncongested, call processing may allowcalls to enter the network and throttling mechanisms may be taken off ofthat path.

FIG. 26 is a flow chart of an exemplary process for performingcongestion control for calls coming through an IP trunk to a line. Theprocess 2600 starts at step 2602. At step 2604, an incoming call isreceived on an IP trunk. At step 2606, a determination is made if thecall is allowed, where a call may not be allowed if an improper messageis received, for example. At step 2608, determination of the call beingallowed is performed. If the call is not allowed, then a “reject”message is sent to the originating call device at step 2610 and theprocess ends at step 2612. If the call is allowed, then the processcontinues at step 2613, where a request for routing translations from acall controller is made. At step 2614, the determined routing andterminating line information is received. Additionally, node segmentassignment information for the originating trunk and terminating lineinformation including all node segment assignment information may alsobe received.

At step 2616, a determination is made as to whether the terminating lineis available. If not available, then at step 2618, a “reject” message issent to the originator and the process ends at step 2620. If at step2616 it is determined that the terminating line is available, then atstep 2622, the network segment status table (e.g., TABLES V-VI) may beaccessed to find usage status for the originating trunk. Results fromthe network segment status table for the originating trunk are receivedat the call controller at step 2624, and a determination of the largestsegment status is made at step 2626 to determine a worst parameter ofthe trunk.

At step 2628, a determination is made as to the status of theoriginating segment at the trunk. If the status is determined to benormal, then the process continues at step 2630. If the status of theoriginating segment is determined to be impaired, then the processcontinues at step 2632. If the status of the originating segment isdetermined to have congestion, then the process continues at step 2634.It should be understood that the status may have more or fewer levelsthan those presented herein. The levels (i.e., normal, impaired, andcongested) represent a range of values determined from networkperformance information reported to the CCM and stored in a table ascollected by network communications devices or nodes on the packetnetwork.

If a determination is made at step 2628 that the status of theoriginating segment at the trunk is normal, then the process continuesat step 2636 (FIG. 26B), where a request to access the network segmentstatus table to find usage status of the terminating line is made. Atstep 2638, results from the network segment status table for theterminating line is received. The termination of the network segmentstatus of the terminating line is performed at step 2640. At step 2642,a determination is made as to the status of the terminating segment. Ifthe status of the terminating segment is determined to be normal, thenthe process continues at step 2644, where an invite is sent for set-upwith a normal or conventional request to the terminating trunk. At step2646, the process continues normal call control and the process ends atstep 2648.

If, at step 2642, a determination is made that the status of theterminating segment is impaired, then the process continues at step2650, where CODEC capabilities of the terminating line are checked. If alower rate CODEC is available, as determined at step 2652, then theprocess continues at step 2654, where a set-up with the lower speedCODEC is sent to the terminating line. At step 2656, the call controllerwaits for a subsequent message from the terminating line, and, uponreceiving a response from the terminating line indicating that the lowerrate CODEC is available at step 2658, a message is sent to theoriginating IP trunk with the new lower rate CODEC information at step2660. At step 2662, the call control process continues to complete callset-up and ends at step 2664.

If at step 2642 a determination is made that the status of theterminating segment is congested or no lower rate CODEC is available atstep 2652, then the process continues at step 2666, where a “reject”message is sent to the originating call device. The process ends at step2668.

Returning back to step 2628, if the status of the originating segment isdetermined to be impaired, then the process continues at step 2632,which enters the process at step 2650 in FIG. 26B. If at step 2628 thestatus is determined to be congestion, then the process continues atstep 2634 in FIG. 26C. At step 2670, an instruction to the callprocessing to enact normal traffic management tools is made and a“reject” message is sent to the originating call device. The processends at step 2674.

In summary, the process provided in FIG. 26 attempts to improve callquality in the packet network when calls enter the network via a trunkto a line in the packet network by examining node segment performancefor the node segments over which the call is to be routed. The CCM mayexamine tables that include node segment status and, if a node segmentis found to be impaired, attempt to lower the rate of a CODEC throughwhich the call is routed. Otherwise, the call may be dropped.

Data Routing

The network performance information may include information indicativeof a node segment being impaired or congestion to the point thatnon-real-time information is buffered, blocked or otherwise impedingreal-time content from being timely communicated therethrough. The node,layer 2, or above protocol stack such as the Multi-Protocol Label Switch(MPLS) Label Description Protocol (LDP) stack, may determine that thenode segment, such as a router, is being overloaded with non-real-timecontent and cause the node to slow down, delay, stop, or drop thenon-real-time content from being communicated through the node segment.The higher protocol stacks may use the transmission state information tomake decisions for Label Switched Paths (LSP) to modified, rerouted, orshaped based upon the link state measured for both real, andnon-real-time content. Once the higher protocol stacks have thereal-time information, functions, such as choosing LSPs or loadbalancing are possible. Oversubscription rules may also be dynamicallycalculated based upon an amount of real-time traffic traversing over apath or segment, utilization and performance information communicated tothe higher protocol stacks, and provisioning engines associated with thehigher network protocols. Given higher stack protocols, such as MPLS, orProvider Backbone Transport (PBT), traffic engineering may be used tosetup and reroute virtual circuits knowing the amount of real-timebandwidth usage and a path state that enables a higher reliability sothat failovers will not exceed oversubscription parameters. This stateknowledge may be used by packet mesh networks where multiple paths existand each path has multiple backup paths. In general, data networks use a1:n path protection schema. When three or more links exist, protectionis typically non-linear as potential bandwidth usage is a function ofthe destinations identified in the routing tables. To enable packetfailover in a 1:n configuration where the amount of real-time traffic isknown provides a network carrier with greater service assurancereliability and metrics to manage the network. In summary, the networkperformance information for segments is stored at network nodes trackingthe real-time bandwidth usage and other performance data. The storednetwork performance information is made available to the higherprotocols, such as MPLS, LDP, and EMS systems to track the amount ofreal-time or near real-time bandwidth being used. Tracking the real-timebandwidth usage enhances network management for provisioning systems,failover protocols, traffic management analysis, and billing systemutilization.

In one embodiment, a decision as to which real-time content ornon-real-time content to prioritize, slow, throttle, block, rate,re-route, or otherwise control may be made based on both networkperformance information and service level commitments or guarantees ofthe quality of service that have been made to a particular customer. Forexample, such decision may be made to minimize the amount of servicelevel credits that have to be made to a particular service provider'scustomers based on how such decisions would impact the ability of theservice provider to satisfy one or more such service levels or qualityof service guarantees. If customer quality of service levels andguarantees are to be used for managing network performance, then adatabase including customer quality of service and other servicecontract parameters may be stored and accessed to verify that thenetwork performance information meets the contractual requirements forcustomers of the communications carrier. In one embodiment, adetermination may be made that a particular application is utilizing toomuch bandwidth through a node segment. For example, an application forstreaming a movie, television show, or other entertainment content maybe utilizing bandwidth at a network node that is being strained todeliver real-time content during a particular time period. Thenon-real-time content associated with that application may be sloweddown, dropped, or rerouted to another node segment so that the real-timecontent being communicated over the node segment may be properlyserviced. The CCM may additionally track applications over time todetermine that other provisioning may be utilized for that applicationduring certain time periods or permanently due to increased traffic,either real-time or non-real-time content, via one or more networknodes.

Enhanced Messaging Services

An Element Management System (EMS) may be used by communicationscarriers to monitor and manage performance of their respective networks.Network performance information may be collected and sorted in a mannerto provide for reporting, provisioning, billing, and troubleshootingpurposes. The functions may use the network performance information anddistinguish between real-time and non-real-time content communications.

FIG. 27A is an illustration of an exemplary network system 2700 thatincludes two networks 2702 and 2704 operated by different communicationscarriers. Each of the networks 2702 and 2704 may be used for providingcommunications services for customers of the respective carriers. In oneembodiment, the carriers are telecommunications carriers. Alternatively,the carriers may provide Internet services or other networking servicesand use equipment that collects network performance informationindicative of performance of the network in communicating real-time andnon-real-time content over the respective networks 2702 and 2704. Thenetwork equipment may be configured to use PIP packets for generatingand collecting the network performance information.

One or more performance data collectors (collectively performance datacollector) 2713 may be configured to be in communication with networkequipment that operates on the network of a carrier, such as network2702. As shown, the performance data collector 2713 is in communicationwith end-point devices, such as IP service point 2706,network-to-network interface 2708, and customer access device 2710, forexample. However, other network communications devices may also be incommunication with the performance data collector 2713, either directlyor indirectly. In one embodiment, the performance data collector maycommunicate with the network communications devices via out-of-bandcommunications paths 2712 a-2712 n (collectively 2712). Alternatively,the performance data collector 2713 may communicate with the networkcommunications devices via in-band signaling paths (not shown).

A performance data manager 2714 may be configured as one or morecomputing devices and be in communication with the performance datacollector 2713. Although shown as two or more separate devices, theperformance data manager 2714 and performance data collector 2713 may beconfigured as a single computing device. The performance data manager2714 may further be in communication with a database server 2716,optionally configured in multiple devices, that is operable to store oneor more databases 2717, including the network performance informationcollected from network communications devices on the packet network 2702by the performance data collector 2713. The databases 2717 stored in adatabase server 2716 may be managed by an off-the-shelf database system,such as an Oracle® database or any other commercially availabledatabase. Alternatively, the database may be created and managed by acommunications carrier or other entity.

In operation, the performance data manager 2714 is configured toinstruct the performance data collector 2713 to request and accessnetwork performance information from network communications devices onthe packet network 2702. The performance data collector 2713 may, inturn, issue requests or polls to the desired network communicationsdevices, either directly or indirectly, to obtain network performanceinformation desired by the performance data manager 2417. In oneembodiment, the performance data manager 2714 may issue commands to theperformance data collector 2713 on a periodic basis (e.g., every 15minutes). More particularly, the performance data manager 2714 may beconfigured to request certain network performance information more oftenthan other network performance information. For example, transmissionquality and connectivity may be collected every second or minute whiletransmission rate and bandwidth is collected every 15 minutes.Alternatively, the performance data manager 2714 may be synchronizedwith the modified Y.1731 stack bins in requesting counter values in eachbin at the appropriate time intervals. Still yet, the performance datamanager 2714 may be configured to request network performanceinformation in response to an event after parsing and examining networkperformance information previously collected. In one embodiment, theperformance data manager 2714 operates to collect data from a datapacket of a single carrier. Alternatively, the performance data manager2714 may be configured to collect network performance information frommultiple networks of multiple carriers, if such permission is providedby the different carriers. The performance data manager 2714 may bemanaged by a carrier or a third party, where the third party isindependent from the carriers and has permission to access and managecertain or all network performance information post-processingoperations for the carriers. In these later two cases, where a thirdparty is involved, quantity of access requests and/or information maybecome a basic billing element used in providing access to thisinformation.

In accordance with the principles of the present invention, theperformance data manager 2714 and performance data collector 2713 may beconfigured to request and receive network performance information,including performance and utilization, associated with communications ofdata packets including real-time and non-real-time content. Theperformance data manager 2714 or performance data collector 2713 maystore the network performance information in the databases on thedatabase server 2716, as distinguished by the different types of contentbeing communicated on the data packet network 2702. It should beunderstood that if other types of content were communicated over thepacket network 2702 and identified as a particular data type (e.g.,video, music), network performance information indicative of theparticular data type may be similarly collected and stored, accordingly.Because the network performance information is stored in a manner thatdistinguishes network performance and utilization for communication ofreal-time content and total content, the service provider, its partners,and customers may use the network performance information to managenetwork communications equipment, monitor network usage, generatereports, and provide billing based on real-time and non-real-timecontent communications over the network.

Collection of the network performance information may be directly orindirectly communicated from each individual network communicationsdevice on the network 2702 or from a table or other repository of a callcontrol manager (e.g. CCM 1502 of FIG. 15) or other device that hascollected some or all of the network performance information desired bythe performance data manager 2714. In one embodiment, when theperformance data manager 2714 instructs the performance data collector2713 to collect the network performance information from the networkcommunications devices, counters are read to collect their currentvalues. For example, the modified Y.1731 counters configured as bins fordifferent time periods over which the counters are used to count thereal-time and total data packets being communicated to and from thenetwork communications devices. In response to the counters performanceinformation being collected by the performance data collector 2713, thecounters within each of the network communications devices may be resetso as to avoid rollover of the counters, a mathematical situation thatis inherently more difficult to manage. Furthermore, tables of thenetwork performance information that are stored at the networkcommunication devices may be cleared or otherwise archived at thenetwork communications devices in response to the performance datacollector 2713 retrieving network performance information from thetables. Real-time archiving of all collected information from the deviceto the databases 2717 may be required to facilitate security or otherbusiness purposes.

The databases 2717 stored in the database server 2716 may be organizedin a variety of ways to enable the network performance information to beprocessed and used for a variety of functions, including billing,reporting, provisioning, generating alerts, managing networkcommunications devices, or otherwise. TABLE IV, presented hereinabove,is an exemplary table of network performance information that may bestored in the databases 2717 in the database server 2716. It should beunderstood that other network performance information may be stored inthe databases to provide additional visibility into the networkperformance at each node segment. Still yet, it should be understoodthat virtually any network performance information that can be collectedby network communications devices may be collected and stored in thedatabases 2717 on the database server 2716.

The database may be further expanded to include statistical or otherinformation derived from the network performance information or otherdatabase systems and/or database information. For example, trends, suchas usage over a time period of an hour, a day, a week, a month, or ayear may be stored in the database in association with each node segmentor otherwise. For example, customer information, circuit IDs, or othermay be stored. The network transmission information and statistics maybe configured to accommodate any billing or post-processing operations.For example, if the principles of the present invention provide forcharging customers differently for real-time bandwidth and non-real-timebandwidth usage, that information may be separately determined andstored in the databases 2717. The databases 2717 may include virtuallyany data structure to accommodate current cost, and pricing structuresassociated with real-time and non-real-time content usage. The currentcost may be defined for consumer, commercial and/or wholesalesubscribers or on a customer-by-customer basis, for example.

The database 2716 server may be configured to enable access to thenetwork performance information stored in the databases to variousentities, including, but not limited to, web entity 2718, user entity2720, billing entity 2722, and operations entity 2724. Each of theseentities may access the network performance information stored on thedatabase server 2716 via a communications device, such as a personalcomputer, mainframe computer, wireless device, or otherwise. Anotherembodiment may include pushing portions of this data from the databaseto similar entities, paging/text terminals, and other alarming andalerting entities.

Web Entity

The web entity 2718 may utilize an Internet interface for displaying thenetwork performance information, as well as customer billing planinformation that distinguishes between real-time network performanceinformation and non-real-time network performance information stored inthe databases 2717 web interface. FIGS. 28A and 28B (collectively FIG.28) are screenshots of exemplary web browser interfaces 2800 a and 2800b, respectively. In web browser 2800 a, an exemplary customer billingplan table 2802 may be used to display a customer billing plan thatincludes usage allocation 2804 and billing rates 2806 associated withthat usage allocation. The billing rates and other billing relatedinformation may be stored on the billing party computer, server hostingthe website, databases 2717, or other server. In one embodiment, theusage allocation may include bandwidth, peak (megabits per second),access time for “anytime” minutes, and access time for daytime minutes,for example. In another exemplary embodiment, the billing rates 2806 mayinclude parameters, such as bandwidth, peak rate, access time (anytimeon a per minute basis), access time (daytime on a per minute basis), andtotal data (on a per one hundred megabit basis). As shown, real-time andnon-real-time settings may be different as network performanceinformation is available for both real-time and non-real-time usages. Itshould be understood that total usage could also be shown or shown inplace of the non-real-time column and the non-real-time informationcould be derived by subtracting real-time content network performanceinformation from the total information. In addition, the information maybe itemized into directional information showing the same types ofinformation identified as information into or out of the customerlocation.

FIG. 28B shows a web browser interface 2800 b includes exemplary table2812 that shows customer actual usage parameters 2814. These parametersreflect the real-time and non-real-time content network performanceinformation collected by the performance data collector 2704 (FIG. 27A)and stored by the performance data manager 2714 in the database server2716. It should be understood that other parameters that distinguishbetween real-time and non-real-time content usage may be utilized forbilling customers. It should further be understood that parameters thatdo not distinguish between real-time and non-real-time usage of thepacket network may be used for billing purposes as well.

Although the tables shown in the web browsers 2800 a and 2800 b showinformation associated with billing, it should be understood that othernon-billing information may be displayed in a web interface. Morespecifically, in addition to the usage information, other information,such as service agreement terms, including quality of service,guaranteed bandwidth, base subscription fees, or any other terms orconditions between a carrier and customers, partners, other carriers, orother commercial or governmental entity may be stored and presented onthe web interface. It should further be understood that the webinterface may enable other, non-subscriber partners to access variousinformation stored in the database. For example, a partner, such as alocal service provider, or other communications carrier may have accessto certain network performance information that the carrier who owns thenetwork performance information may wish to share. For example,transmission connectivity of a network-to-network interface thatcommunicates directly with the other carriers' network-to-networkinterface may be shared. A permissions database or table and associatedsecurity constructs, such as authentication, may be managed by thedatabase server 2716 or other device that define the data that thecommunications carrier is willing to share with other carriers,customers, equipment manufacturers, or otherwise. The permissions tablemay provide different levels of information to different entities.

User Entity

The user entity 2720 may be a user of the communications carrier whomanages the database. The user 2720 may access the network performanceinformation stored in the database 2717 and also perform various othermanagement operations on the databases 2717. For example, the user 2720may generate additional tables, reconfigure the tables, design newdatabase architectures, and so forth, so that network performanceinformation may be expanded and provide customers, partners, vendors,etc., with different or more detailed information, for example. Inaddition, the user may generate different ways of managing the networkperformance information, such as generating statistics based on themodified Y.1731 counter bins, setting up thresholds to cause eventmessages for alerts to be created, setting up and initiating polls tonetwork communications devices for various event-driven ornon-event-driven reasons, and adding statistics processing for thenetwork performance information to provide additional information tomanagement of the communications carrier, customers, vendors, etc. Itshould be understood that the user 2720 may perform any other databasemanagement operation for which the user has proper administrativepermissions to manage the real-time and non-real-time networkperformance information as understood in the art.

Billing Entity

FIG. 27B is an illustration of an exemplary billing entity system 2722for use in determining billing for customers and partners of acommunications carrier. The billing entity 2722 includes a processingunit 2726 that may include one or more processors. A memory 2728 may bein communication and used for storing data and program instructionsduring processing operations. A storage unit 2730 may be incommunication with the microprocessor 2726 and be used to store one ormore databases or other storage repository that include networkperformance information, information derived from the networkperformance information, and billing information. Input/output (I/O)ports 2732 may be in communication with the processing unit 2726 and beconfigured to communicate over a packet network using one or morecommunication protocols as, too, may be the processing unit 2726. TheI/O ports 2732 may be virtual in nature. For example, the I/O ports 2732may operate as an Internet protocol socket or otherwise.

The billing entity system 2722 may use programs for managing andpreparing bills. The programs may include data collection programs 2734,billing programs 2736, and database programs 2738. These programs2734-2738 may be executed by the processing unit 2726. The datacollection programs 2734 may be configured to communicate with one ormore network communications devices in a virtual call path. Thecommunication with the network communications device(s) may transfer raw(e.g., uncompressed) data records between the network and/or end-devicesand the billing entity system 2722. The communication transfer may beinitiated by either the billing entity system 2722, considered as an“information pull,” or by network communications devices, considered an“information push.” The remote network device may contain storage toaggregate multiple records and programming logic to clear informationfrom the device once the raw information transfer has occurred.

Billing programs 2736 may use the raw data records contained within adata packet (e.g., PIP data packet) and parse data fields, such asconcatenated data fields, contained within the received data packetsinto individual raw data fields. Each individual raw data field may beutilized by the database programs 2738 for storage in a database.

The billing programs 2736 may further routinely process the databaserecords. This processing may include consolidation of multiple raw datarecords into one or more processed records, summation of real-timeand/or non-real-time raw data field information into totals and/orsub-totals over a time-window or session duration. These totals orsub-totals may include start and stop time of usage, summation of timeof usage, total packets sent/received with and/or without error,statistical performance calculated values, and/or any other types ofinformation that can be derived via processing raw data records ofnetwork performance information. Additionally, the billing programs 2736may perform ratings, which are the monetization of billing records.Totaled or derived fields may be assessed against a set of businesscharging rules and a monetary charge amount may be established for eachdata record stored in the storage unit 2730 by the database programs2738. The billing entity system 2722 may consolidate multiple ratedbilling records on a per customer basis. By consolidating the multiplerated billing records, additional calculation or rating function mayprovide specific business functions, such as discounting or otherwise.

Continuing, the billing entity 2722 may use the network performanceinformation stored in the database to provide for billing plans forcustomers and other carriers to include billing for both real-time andnon-real-time network usage. This additional resolution of billing(i.e., real-time usage billing) is a result of being able to determinepacket communication of real-time content over the packet network byusing performance information packets, for example. Consumers may bebilled for real-time content usage, non-real-time content usage, andtotal usage of network communication capacity. The capacity may be afunction of the bandwidth usage for real-time content and total packetcommunications over the network. In one embodiment, Erlangs, which isgenerally understood to be mean total traffic volume over a period ofone hour or 3600 seconds (centum call seconds), may be used as a measurefor the carrier to provide accurate billing for customers. The specificcalculation of Erlang may vary to account for different networkperformance information being used to determine the number of Erlangsused during a billing cycle. In accordance with the principles of thepresent invention, the Erlang measure may be used to determinereal-time, non-real-time, and total usage by a subscriber or othercarrier by calculating total traffic volume of subscribers of othercommunications carriers communicating on the communication carriersnetwork in a roaming situation. In addition, because a communicationscarrier may monitor bandwidth and other network performance informationfor both real-time and non-real-time content communication, thecommunications carrier may add or offset a subscribers bill based onfactors, such as transmission quality, connectivity, or rate or othernetwork performance information and/or business purpose, such as aService Level Agreement, that may be collected during a billing cycle.Such offset may also be utilized for other carriers' bills as well.

As an alternative or complement to using Erlangs as a standard ofmeasure, the carrier may assign points or other units of measure to asubscriber for real-time usage and non-real-time usage. For example, areal-time usage minute may be worth three points and a non-real-timeusage minute may be worth one point. The billing may indicate the numberof points that the subscriber has used and charge the subscriberaccordingly. For example, if the subscriber uses thirty minutes ofreal-time usage, which translates to ninety points, then that subscribermay be charged differently from a subscriber using thirty minutes ofnon-real-time minutes, which is only thirty points. Other creative waysof billing based on real-time usage resolution may also be utilized inaccordance with the principles of the present invention. Furthermore,because the database of network performance information may includetimestamps with collected usage information of a subscriber and othercarriers, the billing entity 2722 (FIG. 27A) may use that informationbased on a time of day to set rates during peak and non-peak networkcongestion time periods. This time of day or network congestion timeperiod may be utilized by the carrier to bill the customer for usageduring peak and non-peak times.

The billing process may further use terms of a customers plan to limitnetwork usage for real-time and non-real-time communications. If, forexample, a customer has a service agreement for two thousand minutes ofreal-time minutes, the performance data manager 2714 may monitor acustomer's real-time content minute usage, optionally as measured interms of Erlangs or bytes of real-time traffic, and determine that acustomer has exceeded the limit based on the customer's usage plan. Inresponse to the customer exceeding the usage minutes in his or her usageplan, the carrier may perform a number of different options, including(i) shutting off the users real-time content communications, (ii) allowthe customer to continue using the network for real-time contentcommunications, for example, but use a “best efforts” process or lowestavailable CODEC for allowing access to the network, where “best efforts”means that the user will receive a lower priority status, such asnon-real-time data communications access priority, (iii) premium billthe client so that the client pays extra to continue having priority forreal-time content communications, (iv) trade units, such as allowing thecustomer to use additional non-real-time units for real-time usage at ahigher exchange rate (e.g., five non-real-time usage points for everyminute of real-time content communications usage), (v) take anadvancement towards next month's usage minutes, or (vi) any other planthat enables the user to continue with real-time usage or non-real-timeusage over the usage plan limits. In one embodiment, a message may besent to the customer to select an option for continued service above hisor her service agreement limits. In another embodiment, a customer may“pre-pay” for real-time units, and be denied service once the units areused. In yet another embodiment, two carriers may make business andconnectivity arrangements to inter-exchange database information toallow a subscriber to “roam” onto another provider's network and stillhave access.

The same, similar, or different billing arrangements may be utilized fordetermining billings for commercial entities, such as reciprocal billingbetween carriers based on the real-time bandwidth transmission from onecarrier to another or on an aggregate basis, for example. When managingaccounts with other carriers, trade units of usage, including real-timeand non-real-time content usage, may be resolved at the end of a billingcycle. By having real-time and non-real-time content usage information,trading units can become “creative” such that the carriers may eitherbetter balance the usage of each others' networks or gain a businessadvantage by being able to (i) restrict another carrier's usage of thepacket network or (ii) collect additional fees for providing additionalreal-time content usage or non-real-time content usage of the carrier'snetwork. It should be understood that many real-time and non-real-timecontent usage network performance information parameters may be utilizedin determining billing arrangements with subscribers and billing andsharing level arrangements with other carriers having reciprocal billingarrangements.

Reciprocal Billing

Carriers typically have inter-carrier service agreements that enablecommunications from one carrier to be routed over a network of anothercarrier. These service agreements often have reciprocal billingarrangements whereby the amount of usage of a carrier's network isbalanced or paid for at a certain time period against the usage of thatcarrier's network by the other carrier. This enables the carriers tobalance the service payments other carriers based on usagedifferentials. In accordance with the principles of the presentinvention, the carriers may include metrics or parameters that trackboth real-time and non-real-time content communications over each othersrespective networks. Adding resolution to identify real-time contentusage may identify imbalances occurring between carriers (i.e., onecarrier is communicating significantly higher real-time content overanother carrier's network). The CCM or other monitoring device mayrecognize this imbalance and determine that communications tosubscribers may be routed to the network of the carrier that has a highbalance as a credit for communications routed over that carrier'snetwork may exist. A decision may be made to route the communications,real-time or non-real-time content communications or both, over thatcarrier's network.

As another example of carrier level service being imbalanced, carrierservice level agreements may specify a certain quality of service ortransmission rate, possibly with real-time content and non-real-timecontent being separately specified. A carrier may monitor for theseservice level agreement parameters to determine if another carrier ismeeting its obligations under the agreement. If the obligations underthe agreement are not being met, then the service provider may receivecredits toward additional free communications services. Routingdecisions may be made in response to determining that these or otherservice level agreement parameters are not being met and credit isavailable.

Another example of routing decisions being made in response to trackingnetwork performance information or of service level agreementinformation may include monitoring pricing by other carriers throughouttimes of the day that are scheduled or in response to a high demandoccurring within that carrier's network. The other carrier may“advertise” pricing or other parameters, such as bandwidth availabilityat an NNI node, for example, to notify other subscribers of pricingchanges, availability, transmission problems, etc. The CCM orperformance manager 2714, in learning of price changes either upwards ordownwards, of other carriers may make routing decisions based on thosepricing changes. The decision may also include factoring current credit,cost to carrier, customer bandwidth requirements, or any other parameterassociated with performing communication services at a certaintransmission quality, and cost to the carrier. Routing “shopping” may beperformed by collecting such “advertised” information during regular PIPpacket communications or special rate collection requests to eachcarrier network-to-network interface or session border controller withwhich the carrier has a service level agreement.

Operations Entity

FIG. 29 is an illustration of an exemplary graphical user interface(GUI) 2900 that displays a schematic 2902 of a packet network 2904 andperformance monitoring devices 2906. The schematic 2902 is a graphicalrepresentation of network communications devices located on the network2904 including node segments over which real-time and non-real-timecontent communications are communicated. It should be understood thatmore or less detailed schematics of the packet network 2904 or otherexternal devices or networks may be displayed. The network performanceinformation stored in the database 2717 on the database server 2716(FIG. 27A) may be utilized to graphically represent problems and alertson the schematic 2902. For example, if communications on a node segmentis determined to be normal, then a solid line, such as line 2908 may bedisplayed on the schematic. If the network performance informationindicates that a node segment bandwidth is being utilized to either fullor over-capacity, then the node segment may be highlighted, such as nodesegment 2910 using a thicker line than other node segments that areoperating normally. Alternatively, color coding, flashing, or othergraphical representations may be utilized to indicate high trafficvolume. If a node segment is determined to be impaired, then theschematic may show a dashed line, such as line 2912. If the networkperformance information indicates that a node segment has congestion,then the node segment line may be dashed, such as line 2914 beingvisually different from a line indicating impairment. It should beunderstood that other graphical representations indicating high usage ofreal-time bandwidth, non-real-time bandwidth, or any other networkperformance information that is within a range or outside of a thresholdmay be used for graphical notification or alerting a user of an abnormalcondition occurring on the network 2904. Other colors, text, pop-upwindows, or any other graphical features may be displayed on theschematic for normal or abnormal operation of the network. Sounds mayalso be used for notification, alerts, or alarms. In one embodiment, thegraphical user interface may enable the subscriber to position a cursorusing a pointing device, such as a mouse, over a node segment to causenetwork performance information to appear in a pop-up window orotherwise displayed in relation to the node segment. The node segmentinformation displayed may include current and, optionally, historicalnetwork performance information, and be displayed either as a value orgraphically. Notifications may also be displayed in response to cursorpositioning.

FIG. 30 is a screenshot of another exemplary graphical user interface3000 that is displaying a chart 3002 of node segments status usage for aparticular node on a network. Three network performance informationparameters are displayed on the chart 3002, including total usage,real-time usage, and non-real-time usage as shown in the legend 3004.Total network usage is shown by line 3006, which changes over the courseof a day as customers are increasing and decreasing usage of the nodesegment from which the usage data has been collected and stored in thedatabase server 2716 (FIG. 27A). The total usage line 3006 is a sum ofthe real-time and non-real-time content usage. A total usage line 3006is shown to have a morning peak at about 9:00 a.m. at point 3008,mid-day peak at about noon at point 3010, and evening peak atapproximately 8:00 p.m. at point 3012. The real-time and non-real-timeusage lines show that real-time usage increases at various times of theday and non-real-time usage increases at other times of the day. Forexample, the non-real-time usage spikes at about 9:00 p.m. at point 3014when, presumably, customers are downloading movies, music, or otherwiseweb surfing. It should be understood that other graphicalrepresentations may be made of one or more node segments or transmissionpaths through a network. It should also be understood that other typesof network performance information, including derived information (e.g.,trend lines), may be displayed to show transmission quality or othertransmission characteristics or node characteristics at any point withina network being monitored in accordance with the principles of thepresent invention as provided by the performance data manager 2714 (FIG.27A) and use of PIP packets.

In addition to showing the usage information on a chart, alerts, trends,or other statistics may be presented on the GUI or via any otherreporting method. Still yet, reports of the network performanceinformation may be generated through the use of the stored networkperformance information stored in the databases 2717 and provide a userinterface for selecting, sorting, tabulating, and any other functionthat can help a user generate current and historical reports, alerts,alarms, or any other information associated with or resulting fromnetwork performance information collected from a network.

Provisioning Entity

The performance data manager may additionally use the networkperformance information that is collected from the packet network toprovide provisioning functions. Provisioning may include a variety offunctions, including (i) tracking path or element oversubscription ratesand utilization prior to allowing network provisioning to occur, (ii)managing network performance tracking by creating reports for newlycreated entities, (iii) dedicating or calculating failover, (iv) loadbalancing for re-routing of real-time or non-real-time contentcommunications, (v) retrieving and presenting state information onnetwork utilization and available resources to network managers in theform of reports and trend lines to determine where congestion isoccurring, (vi) displaying locations where additional routers, gateways,or other network communications devices may be desired to alleviatecongestion or provide safety valves for network communications devicesthat require higher bandwidth capacity during certain times of the day,or (vii) providing any other network management functionality based onthe network performance information as described herein. In addition,the provisioning may enable automatic response to alerts or warningsthat are detected by the performance data manager on a real-time or nearreal-time basis. For example, if an alert is created by a threshold forbandwidth capacity, the performance data manager may seek to re-routereal-time or non-real-time content communications. Alternatively, if aspike in real-time content communications is occurring at a node thathas non-real-time data being communicated at the same time, theperformance data manager may notify the node to halt new provisioning ornew communications sessions of the data packets including non-real-timecontent until the real-time content communications rate has decreased.The performance data manager 2714 may further be configured to directone or more network nodes to change the bandwidth of a CODEC, closeports, send messages to other carriers to notify the other carrier of anoverload or over-usage condition coming from their network, or performany other provisioning function through the use of monitoring thenetwork performance information, as provided.

The performance data manager 2714 may be configured to automaticallydetect a problem within the network and issue one or more tests, such asa trace route, to be performed on an end-to-end basis. For example, inFIGS. 27 and 29, a message may be sent from the performance data manager2714 via the performance data collector 2704 to cause a test to be madebetween customer equipment 2916 and 2918. The test may include sendingPIP packets for a one minute time period, for example, between thecustomer equipment 2916 and 2918. During that time period, the customerequipment 2918 may collect network performance information, such astransmission quality, transmission rate, and transmission connectivity,optionally as associated with real-time and non-real-time contentcommunications. Even though the customer equipment 2918 resides with adifferent carrier, the network performance information that wascollected from running a test between the customer equipment 2916 and2918 may be collected by the performance data collector 2704 withoutsharing any company specific, sensitive information of the networkcarrier or carriers managing network 2920. In one embodiment, thenetwork performance information collected from the customer 2918 is aresult of “stitching” (concatenation) of network performance informationby appending the information to PIP packet payload as communicatedthrough each of the nodes in the transmission path between customerequipment 2916 and 2918. Alternatively, the performance data collector2704 may request data directly or indirectly from each of the nodesalong the transmission path between the customer equipment 2916 and2918.

Modified Trace Route

Network performance information collected for real-time content andnon-real-time content communications may provide an indication thatthere is a performance problem existing at a node segment within apacket network. A call end-point, CCM, or node within the packet networkmay determine that a problem exists based on the PIP performanceinformation and automatically trigger a path trace route in the PIPpacket flow. Given that higher protocol stacks can move or otherwisealter a packet transmission path without consulting the CCM, thisfunction facilitates identifying nodes and segments being traversed atthe time the trouble is encountered. It should be understood that asidefrom a statically configured PIP data stream for embedded networkequipment, that PIP sessions may be constructed in an ad hoc basis foruse on packet devices not normally associated with that network. In thead hoc PIP case, the user end-point creates a PIP session from and to apoint inside the network provider's network or to the far end-point toprovide the real-time bandwidth and PM data function. An ad hoc PIPpacket flow may be set up with each call from end-point-to-end-point or,alternatively, to an anchor point in the serving network provider'snetwork. Once the PIP and PM stack detect a performance thresholdcrossing, a trace route may be initiated to identify the location withinthe path that is having a problem. A network node element may store thetrace information and make the trace information available directly tothe CCM and/or user. Additionally, the trace information may becommunicated to the call control protocol stacks to be passed back tothe CCM or EMS for troubleshooting. Other information may be stored withthe trace information, such as time, date, session information and soon. The troubleshooting procedure may be performed to isolate the nodesegment having a problem with either or both real-time and non-real-timecontent being communicated through the node segment. The CCM or node mayinitiate a modified trace route to communicate one or more data packets,such as PIP packets to or through the node segment of concern to collectnetwork performance information through that node segment that may behaving a transmission problem. The network performance informationgenerated from the trace route, which may last for one or more PIPpackets being communicated over a long enough duration to determine thenetwork performance information at the node segment of concern. If thenode segment is a node segment located at the edge of another network ortype of network (e.g., network-to-network interface) then networkperformance information collected at the other node in the other networkmay be communicated back to the CCM or originating node with networkperformance information specifically related to the modified traceroute. Other “carrier specific sensitive” network performanceinformation of the other network may otherwise be prevented from beingaccessed by the other network. It should be understood that ad hock PIPpacket flows may also be associated with encrypted path protocols andpresence protocols that establish remote network connectivity, such asPPP, SLIP, and/or other remote agents.

The collected network performance information may enable not onlymessages and alerts to be sent to operations management of the network,but also notify customers, partners, affiliates, and other networkcarriers of problems, congestion, or other situations or events of thenetwork For example, if a determination that real-time contentcommunications usage is high, a notice may be sent to subscribers andother carriers of the situation and notification of increased billingrates. Similarly, if a determination is made of high amounts ofcommunications to other carriers, the carrier may elect to “shop”communications to destinations via other, lower priced carriers. Thecarrier may have thresholds for many terms and conditions of subscribersand partner carriers and automatically, semi-automatically, or manuallymake provisioning changes based on determining that a threshold has beencrossed based on the collected network performance information.

FIG. 31 illustrates an embodiment 3100 of the OSI basic reference modelof networking that include seven different layers. The referencenumerals (3102-3114) on the left side of the model are used to describethese different layers of the reference model. Each of the layersprovides protocols for certain types of operations. More specifically,the seven layers include: physical layer 3102 (Layer 1), data link layer3104 (Layer 2), network layer 3106 (Layer 3), transport layer 3108(Layer 4), session layer 3110 (Layer 5), presentation layer 3112 (Layer6), and application layer 3114 (Layer 7). Typically, physical layer 3102conveys bit streams, such as bits containing electrical impulses, lightor radio signals, through a network at the electrical and mechanicallevel. The physical layer 3102 provides the hardware means for sendingand receiving data, including defining cables, cards, physical aspects,data coding, and medium (B8ZS, DS-3, etc.). At the data link layer 3104,data packets are encoded and decoded into bits. The data link layer 3104further furnishes transmission protocol knowledge and management, andhandles errors in the physical layer, flow control, and framesynchronization, including Ethernet, Frame Relay (FR), ATM,Multi-Protocol Label Switching (MPLS), etc. In the following examples,the network performance information is stored in the data link layer3104, and, optionally, the other layers 3106-3114.

The network layer 3106 provides for (i) switching and routing, and (ii)creating logical paths, known as virtual circuits, for transmitting datafrom node to node within a packet network. The transport layer 3108provides transparent transfer of data between end systems, or hosts, andis responsible for end-to-end error recovery and flow control. Oneexample of a network protocol is Internet Protocol (IP). An example of atransport layer protocol is Transmission Control Protocol (TCP). Thesession layer 3110 establishes, manages, and terminates connectionsbetween applications. The session layer 3110 sets up, coordinates, andterminates conversations, exchanges, and dialogues between theapplications at each end of a network path. The session layer 3110further manages session and connection coordination. The presentationlayer 3112 provides independence from differences in data representation(e.g., encryption) by translating from application to network format,and vice versa. The presentation layer 3112 transforms data into theform that the application layer 3114 can accept. Such presentation layer3112 typically includes text, voice, and video compression. Theapplication layer 3114 also supports application and end-user processes.Some examples of application layer 3114 applications include email andfile transfer applications. Each layer interacts directly with the layerimmediately beneath it and provides facilities for use by the layerabove it. In addition, the protocols on each layer enable entities tocommunicate with other entities on the same layer.

FIG. 32 illustrates an embodiment 3200 of various maintenance entities(ME) depicting defined multiple administrative domains, such asSubscriber Maintenance Entity (SME) 3218, Ethernet Virtual Connection(EVC) ME 3220, Operator ME 3224 and 3226, Network-to-Network (NNI) ME3222, and User-to-Network (UNI) ME 3228 and 3230. The domains have beenconstructed using Maintenance Entity Group (MEG)—8 level structures toprovide limited views into the quantity and types of informationavailable to each level (domain). A maintenance entity is a subset ofall available maintenance data that has been grouped together for accessby a particular network participant, such as a subscriber, Ethernetprovider, network operator, or virtual network operator.

The OSI reference model described in FIG. 31 defines specificfunctionality contained in each of its layers 3102-3114. The principlesof the present invention may utilize Ethernet services, which operate inthe Data Link Layer 3104 of the OSI reference model. The Ethernetprotocol is identified as ETH Layer 3232 in FIG. 32, where FIG. 32illustrates the transport layer 3108 of the OSI model 3100 as TRAN layer3234.

In one embodiment of the present invention, real-time transmissionperformance information acquired in the Data Link Layer 3104 iscommunicated into one or more of Physical Layer 3102, Network Layer3106, Transport Layer 3108, Session Layer 3110, Presentation Layer 3112,and Application Layer 3114. In another embodiment, the real-timetransmission performance information acquired in the Data Link Layer3104 may be communicated into other Data Link Layer protocols, such asATM, MPLS, Frame Relay, or other protocols. This real-time transmissionperformance information may be used to provide real-time notification ofthe ETH Layer 3232. This real-time transmission performance informationmay also be used to complement existing protocols and capabilities toprovide quicker response time to network changes identified to ETH Layer3232.

Data Link Layer 3104 from and to Physical Layer 3102.

In one embodiment, real-time transmission performance informationacquired in the Data Link Layer 3104 may be communicated to the PhysicalLayer 3102. In one embodiment, the degradation of a copper-based linkdue to induced noise or any other source of impairment, delay, or lossof data could limit the quantity of information that can be carriederror-free across the link. The transmission performance informationcarried in the PIP packet is capable of identifying this degradation.This degradation may be reported to the Physical Layer 3102, where aprotocol operating on the Physical Layer 3102 realizes the degradationand modifies the route to optimize throughput and overcome theimpairment, such as rerouting the link to an alternative physical copperlink or a reduction in the number of Quadrature Amplitude Modulation(QAM) windows or change to another transmission schema all together.

Data Link Layer 3104 from and to Data Link Layer 3104.

In one embodiment, the above degradation may be communicated to the DataLink Layer 3104, where the multiplexed protocols of the Data Link Layer3104, operating in parallel with the Ethernet due to physical layermultiplexing and protocol isolation, realize the degradation and modifytheir operation to overcome the impairment by conducting an MPLS FastRe-Route.

Data Link Layer 3104 from and to Network Layer 3106.

In one embodiment, the above degradation may be communicated to theNetwork Layer 3106, which could alter network traffic routing to reroutepackets around the degrading link. This reroute may involve moving thesession from one network operator to another network operator. It shouldbe understood that the principles of the present invention may beutilized with any Network Layer (Layer 3) 3106 protocol, including IPv4,IPv6, or otherwise. It should further be understood that the principlesof the present invention may be utilized with any protocol operating onany other layer.

Data Link Layer 3104 from and to Transport Layer 3108.

In one embodiment, the above degradation may be communicated as roundtrip delay and other parameters to the Transport Layer 3108, where theTCP Sliding window function may be dynamically altered to modify thewindow size, thereby reducing the amount of subsequent retransmittedpackets and avoiding congestion. In such an embodiment, suchcommunication allows the adjustment of the window size sooner thancurrent implementations.

Data Link Layer 3104 from and to Session Layer 3110.

In one embodiment, the above degradation may be communicated to theSession Layer 3110, where the session management functions could modifyschedulers, shapers, or any network element function that provides andcontains the Quality of Service (QoS) parameters, thereby dynamicallyadjusting the quantity of packets in a session. The effect ofdynamically adjusting the quantity of packets in the session is thatcongestion points should experience relief as the quantity of packetsflowing into a network node or element is reduced.

Data Link Layer 3104 from and to Presentation Layer 3112.

In one embodiment, the above degradation may be communicated to thePresentation Layer 3112, where the presentation protocol coulddynamically control a video codec forcing a repeat of the last videoframe or reducing frame quality, frame resolution, frame size, framerate or otherwise.

Data Link Layer 3104 from and to Application Layer 3114.

In one embodiment, the above degradation may be communicated to theApplication Layer 3114, where notification is generated and communicatedto a user indicating that the network is experiencing congestion and tobe patient until the congestion clears, try the communication later, ortry to re-connect using different connection parameters. For example, ifa user is engaged in online gaming, the application layer may notify thegamer that the network is slow and to wait before engaging in a fiercebattle to avoid the network not having enough bandwidth to facilitatethe online action. In another embodiment, the application layer 3114 maydetermine that the user is a low priority and cut or kill the networkconnection to the gamer or user.

Other uses of data packets including being passed between the Data LinkLayer 3104 and Application Layer 3114 may include communications controlto manage multiple real-time sessions when the user exceeds availablecommunications resources. Functions, such as presenting the user withusage statistics of network performance information real-time content(e.g., real-time usage or bandwidth) versus total bandwidth, sessionusage of real-time bandwidth, and the ability to selectively chooseCODECs and session types, such as video phone versus a voice-onlycommunications modes, are enabled utilizing the principles of thepresent invention. Load balancing of real-time traffic when multiplepaths are available may also become user selection modes.

Layers to MEs.

In another embodiment, and continuing with FIGS. 31 and 32, real-timeperformance information may be communicated from the Physical Layer3102, Data Link Layer 3104, Network Layer 3106, Transport Layer 3108,Session Layer 3110, Presentation Layer 3112, and Application Layer 3114into the MEs (e.g., subscriber ME, EVE ME, and NNI ME of FIG. 32). Thisreal-time information can be used to complement ME information, supportreal-time modification of network processes and protocols, and assistdomain administrators in management of a hybrid network or group ofnetworks, such as a Metro Ethernet Network (MEN). Several descriptionsof the use of real-time information flows from protocols of various OSIlayers into MEs are described below.

Further, the Metro Ethernet Network Nodes (see FIGS. 32 and 33) mayutilize the information contained in a PIP packet to actively determinethe best path for each connection within its network. One or morevirtual performance tables (VPTs) (FIG. 34) may be created at the MENnode(s) that inputs information relating to each network node. The MENnodes may determine that a particular link goes down at a particulartime of day, such as in a carrier's maintenance window, and, inanticipation of this event, reroute the data traveling on thatparticular link around it to other links, thereby relieving thecongestion on a particular link or network node. Best path metrics mayalso be used to determine if certain real-time or non-real-time datacontent needs to be held up for a period of time to assist withrelieving the congestion on a particular portion or link of a network.Tables, such as VPTs, may be used by the MEN to anticipate potentialcongestions on a network and proactively reroute the data on other linksto avoid the congestion.

Physical Layer 3234 from and to ETH Layer 3232.

In one embodiment, and continuing with FIGS. 31 and 32, the abovedegradation occurs on the copper link of an NNI ME 3222, a portion ofthe circuit that is not Ethernet-based. In this instance, this copperlink is providing an end-to-end Virtual Ethernet service as the NNIportion of the EVC This degradation may be reported as a change(reduction) in the amount of bandwidth available on the circuit linkfrom the Physical Layer 3102 to the ETH Layer 3232. This information maybe included in the appropriate ME domains by a network element or node,thereby allowing other network elements (upstream and downstream fromsuch network element) the ability to react to the degradation prior tolink failure. Such communication of degradation information provides theability to try to pre-establish an alternative to maintain an end-to-endsession in advance of a failure.

In another embodiment, a “route flapping” degradation by the NetworkLayer 3106 may be reported from and to the ETH Layer 3232. “RouteFlapping” is a common term to describe the recalculation of route tableswithin an network element typically due to a link having marginalconnectivity; i.e., conditions are such that the link may “flap” and bemomentarily considered out of service, then naturally recovering andbeing placed back in service by a network element. This route flappingmay occur many times over a time interval. Each time the network elementis restored, a route table re-calculation may be requested by theNetwork Layer 3106 routing protocol. A network element could include thePIP PM information in the appropriate ME domains, thereby allowing otherNEs the ability to react to the degradation prior to link failure andassess real-time stability prior to restoring the link. A potentialreaction could be to identify an alternative network operator or networksegment, thereby routing around the portion of the network that is“flapping.” Secondarily, threshold information could be communicated toa network of another carrier to allow the other network to react to thedegradation prior to the outage becoming more severe.

TCP Sliding Window

Transport Layer 3108 to and from ETH Layer 3232.

In one embodiment, a reduction in a “TCP Sliding Window” containedwithin the Transport Layer 3108 may be reported to the MEs. Thisreduction would signal to the ETH Layer 3232 that congestion isoccurring somewhere within a virtual circuit over which PIP packets arebeing communicated. The congestion may be in the subscriber's network,where a network operator would not otherwise have visibility. In thisembodiment, the TCP/IP sliding window field is modified in real-time,regardless of any network technologies, thus providing quicker TCP/IPsliding window response to performance issues.

The sliding window field within the TCP/IP protocol is modified toreflect performance changes occurring in the network. This modificationmay occur at any network node anywhere within a communication path. TheTCP sliding window field modification may be accomplished by rewritingthe specific TCP sliding window field within the TCP/IP packet as ittraverses through a network node.

Session Layer 3110 to and from ETH Layer 3232.

In one embodiment, a change in session connection quality by the SessionLayer 3110 to the ETH Layer 3232 may be reported. The Session Layer 3110may notify the ETH layer 3232 that a QoS parameter, has been modifiedthereby dynamically adjusting the quantity of packets in the session. Anetwork operator or Ethernet provider may use the QoS information tomanage other EVCs within an MEG, including Connection Admission Control(CAC).

Presentation Layer 3112 to and from ETH PIP flow on Layer 3232.

In another embodiment, a CODEC buffer management algorithm within thePresentation Layer 3112 may communicate to the ETH Layer 3232. Here, anotification may signal that video CODEC buffers have multiple underflowevents resulting in repeats of the last B-frame in an MPEG-4 video, forexample. Underflow events are indicative of lost or delayed packets. TheETH layer 3232 may use this underflow information to by-pass thedegraded segment by choosing an alternate path.

Application Layer 3114 to and from ETH Layer 3232.

In another embodiment, a user program within the Application Layer 3114could signal to the ETH Layer 3232 that the “Network is Slow.” Thecomplaint may be reacted to by the ETH layer 3232, whereby dynamicidentification of a degraded segment and an attempt to modify thesession path to circumvent the degradation may be performed.

In another embodiment, instead of directly modifying fields withinexisting protocols traversing through a network element at any layer inthe OSI reference model, an alternative may include establishing aVector Performance Table (VPT) within the network element. This VPT maybe created and managed as part of a network element operating system andembedded system programming.

In FIG. 33, an embodiment 3300 of an exemplary network element or nodeis illustrated. In this embodiment, the NE 3302 has four physicalinterface connections 3304-3310 that connect to other NEs (not shown)via connections 3318-3324. In addition to the physical interfaceconnections 3304-3310 to other NEs, internal interconnections betweenphysical interfaces exist. An example of an internal connection isreferenced as 3312. In one aspect, the NE 3302 further includes aprocessor 3312 and memory 3314 in accordance with that described herein.Although not shown, physical or virtual internal connections may existas point-to-point, point-to-multipoint, or multipoint connectionsbetween any or all physical interface connections 3304-3310 on a perpacket basis. Many currently available NEs may provide differentinternal connection paths, which may result in different packetperformance on a per packet basis. These different performances may be aresult specific to internal packet handling processes, such as differenttypes of queuing, scheduling, and rate shaping among other packetprocess handling. A particular path through a network element may yielddifferent performance measurements than other paths. Other internalarchitectural structures could be present, too, that may impact internalperformance of a network element.

FIG. 34 illustrates an embodiment of exemplary virtual performancetables (VPTs) 3402 a-3402 n (collectively 3402). The VPT functionalitymay be predetermined or operator defined and configured via the embeddedprogramming on the NE. The NE vendor may allow the operator thecapability to dynamically size the VPT via configuration parameters.Within an NE, bi-directional ME performance information is captured andplaced into the VPT 3402 a in the “current” timestamp 3412. At a latertime interval defined as delta t (Δt), the information contained in timestamp 3412 for VPT 3402 a is moved to VPT 3402 b; the informationcontained in time stamp 3412 for VPT 3402 b is moved to VPT 3402 c, andso forth. Alternatively, a new VPT may be created and VPT 3402 a maysimply become 3402 b as a result of the new VPT being created. Theinformation may also be placed in bins or memory locations or added,summed, averaged, or otherwise summarized or used in calculations, theresult of which is placed in such bins or memory locations. Bins, suchas modified Y.1731 bins, may be associated with time intervals, MEs,levels of access, operator identifiers, or other parameters used toidentify, communicate, process, collate, or allow access to informationincluded in the bins. Bins that collect network performance informationover shorter time intervals may be periodically added into bins thatcollect network performance information over longer time intervals. Oncethe “current” time stamp is empty, bi-directional ME performanceinformation is placed into the “current” time slot, as illustrated inVPT 3402 a. In essence, a first-in, last-out VPT queue is established.However, other temporal related VPT configurations may be utilized inaccordance with the principles of the present invention.

The table size of the VPTs 3402 may be a function of the quantity ofmemory allocated, the types and quantities of network performanceinformation captured. The network performance information may includelink number 3414, real-time and total bandwidth usage, packet loss 3416,latency 3418, jitter 3420, delay 3422, real-time application data,non-real time application data, total data, and the Δt or time stamp3412 between successive samples. Also, “stitched” network performanceinformation (i.e., network performance information from other networkelements) for each NE could be included in the VPT 3402. In one, the VPT3402 could be dynamically sized to accommodate the data.

FIG. 35 describes an exemplary maintenance entity data packet or logicalpacket flow through a network entity 3502. Bi-directional data packetflows carry network performance information. FIG. 35 further illustratesingress data packet flows 3508 and 3514 and egress data flows 3510 and3512 through the NE 3502. Within the NE 3502, the ME embeddedprogramming determines local NE performance measurements and attachesthis information on the end of the ME or network performance informationpayload portion of a packet, as discussed further below. The payload maybe encapsulated within the envelope of the Ethernet Protocol or PIPpacket format. In one embodiment, this information could be encapsulatedwithin additional layers of higher protocol information, such as TCP/IPpacket protocols.

An exemplary logical structure of the payload portion of the PIP packetis described in FIGS. 36-39. FIG. 36 illustrates an exemplary PIP packetpayload ingress flow in direction 1. FIG. 37 illustrates an exemplaryPIP packet payload egress flow in direction 1. FIG. 38 illustrates anexemplary PIP packet payload ingress flow in direction 2. FIG. 39illustrates the PIP packet payload egress flow in direction 2. In thesedata flows, the addition of the NE 3502 performance information at theNE 3502 egress in each direction is shown. This information may beappended to the payload of a packet received on the ingress as the flowis processed through the NE 3502. An end-station, the last deviceparticipating in the PIP packet process, may collect all NE performanceinformation from each NE in a communication path, as illustrated in FIG.40, which depicts PIP packet payload data flows of the end-station(i.e., data flows to and from the end-station). In one aspect, there aretwo directional paths, as the circuit is full duplex (i.e., transmittingand receiving in both directions concurrently, sometimes on separatephysical facilities. Since the call path in each direction could bedifferent and subject to differing forces that modify the performancestatistics, two bi-directional flows may be used. The performancestatistics at each end of the PIP flow may be concatenated andtransmitted to the far end so each end of the transmission orcommunication path holds both the transmit and receive networkperformance information data.

This VPT information may be used locally via new protocols operatingoutside the PIP packet or with modifications to existing protocols toallow the use of VPT information. As described above, in one embodiment,rather than directly writing specific network performance informationinto other OSI Layer protocols, the network performance information maybe made available via the VPT 3402. Each OSI Layer's protocols mayreference any network performance information to make enhancedoperations decisions.

The VPT 3402 enhances current data flows by capturing not only currentdata flows, but also providing historical captures over defined timewindows that are nΔ samples deep. The additional samples can enablepredictive functions, which can be used to improve the reliability andavailability of the session or user experience, perform networkmaintenance, provision new network hardware or media, design new networkconfigurations, or enhance inter-network communications.

The VPTs 3402 (FIG. 34) may be extended to include the collection of VPTnetwork performance information across a single operator or multipleoperators. Conceptually, this collection is illustrated in FIG. 41,which depicts an embodiment 4100 of a Vector Performance CorrelationEngine (VPCE) 4102. Individual VPTs 4104 a-4104 n (collectively 4104)from a network element may be communicated via in-band or out-of-bandcommunication links 4106 a-4106 n (collectively 4106) to the VPCE 4102.These VPTs 4104 could be transmitted as encapsulated information usingcommon protocols, such as TCP/IP. The entire set of VPTs 4104 includingnΔt performance samples could be sent by the NE or polled via the VPCE4102. Alternatively, each current sample of VPTs 4104 may be sent orpolled and the VPCE 4102 may be utilized to establish and maintain ahistorical database of the performance samples for each NE.

Once the network performance information is gathered at the VPCE 4102,the network performance information may be processed to provide anencompassing performance management view of one or more networks basedon input from each NE. This centralized network performance informationstore may be used in a variety means such as, but not limited to,Service Level Agreement (SLA) validation, near real-time NE management,predictive network management, and other functions.

Customizable algorithms and calculations that use the current andhistorical network performance information may be developed and includedas part of the embedded programming of the operating system of the VPCE4102. VPCE 4120 may include memory 4108, one or more processors 4110,which may include cell processors having two or more processors on asingle chip, one or more databases 4112, and one or more I/O ports 4114.The algorithms and calculations may be performed using these computingresources contained within the VPCE 4102. Information processed withinthe VPCE 4102 could be made available to other network systems (notshown), such as a multimedia Call Control Manager (CCM) or other networkmanagement systems using the I/O ports.

In addition, the VPCE 4102 may use the data contained in the VPTs 4104as historical logs for determining when the performance of a certainlink 3414 in a network experience failure or deterioration due tocongestion or other technical problem The information contained in thePIP packets may contain the historical data rate performance asdiscussed herein showing the network nodes and links and based on datacontained in a particular VPT 4104, such as timestamp 3412,determinations can be made that a particular node or link sufferstechnical problems, such as congestion during specific times of the day.The VPCE may also determine the gapping between calls based on thishistorical network performance information contained in individual ormultiple VPTs 3402.

In one embodiment, the VPCE 4102 correlated information is used tocreate a near real-time exemplary Graphical User Interface (GUI) 4202 asillustrated in FIGS. 42 a and 42 b, which is an illustration of anembodiment 4200 of such a GUI. In these figures, possible connectionpaths may be illustrated as links 4204 a-4204 n (collectively 4204) thatconnect NE 4206 a-4206 n. The links 4204 n, 4204 a, 4204 g, 4204 e, 4204c, and 4204 d are being used to support end-to-end connectivity. Link “. . . ”, 4204 b, and 4204 f are alternative circuits available tosupport connectivity, but are currently not carrying traffic. Path 4204e may change color, such as from yellow to red, indicating the link isin severe trouble or congested. The width of the line representing 4204e has been reduced to indicate reduced packet flow. Wider lines mayrepresent greater packet flow. One color may be used to representreal-time application packet flow and another color for non-real-timepacket flow within a single path as illustrated. Based upon this visualnotification, the NMS Operator may take steps to route traffic currentlytraversing links 4204 e to 4204 b or 4204 f. Alternatively, thesechanges could be performed automatically as described above. The same orsimilar graphical user interface 4202 can be provided for transmissionpath segments of a packet network using stitching and communicating theVPT or network performance information through in-band signals to theend customer where the information may be displayed to detail thenetwork performance behavior of the packet transmission paths that thecustomer is utilizing or being sold. This principle of communicatingnetwork performance information across networks of customers can also beapplied ad hoc without the knowledge of the operator withNetwork-to-Network interfaces at the boundaries of the 3^(rd) partyservice provider to provide each end-point with the network performanceinformation of each network segment. This “man in the middle” scenarioenables tracking of the real-time bandwidth transmission characteristicsof customers or other third parties, along with the other PM data. Theboundary or segmentation principle can also be utilized across wirelesstechnologies, whereby multiple wireless connectivity segments areavailable. In this case, the PIP and VPT tables would provide PM ornetwork performance information about the wireless RF route performance.It should be understood that using this boundary principle can apply toany technology deployed between two MIP or MEP points.

In yet another embodiment, one or more links 4204 may further includeindicia representative of the quantity or percentage of real-timeapplication packet flow versus non-real-time packet flow. FIG. 42Crepresents an enlarged view of an exemplary link 4204 i that displaysthe amount of real-time application flow relative to the amount ofnon-real-time application flow by showing two different types ofindicia, in this case rectangles, relative to each. For example, indicia4208 may include a different color, cross-hatching, shading, shapes, orother type of indicia that is different than that for indicia 4210. Inthis example, indicia 4208 indicates the amount of non-real timeapplication flow and indicia 4210 indicates the amount of real-timeapplication flow. Further, the general dimensions, such as widths of theindicia 4208 and 4210 may reflect the application flows relative to eachas well. In FIG. 42C, the amount of non-real time application flow isshown as being less than that for the real-time application flow by theindicias 4208 and 4210 having both different hatching and widths. Anyindicia and dimension of indicia may be used to readily present thisinformation to a user.

In another embodiment, information exchange between the ETH Layer 3232to and from and the OSI layers 3102, 3106-3114 may manifest itself inopen and closed loop interactions. Open and close loop systems are welldefined in Modem Control Theory text books. In summary, in an open loop,information is exchanged without a feedback loop to track a response tothe information. In the closed loop manifestation, feedback loops arepresent, thereby providing dynamic control of the response to theinformation. The principles of the present invention can use either openor closed loop manifestations.

FIG. 43 illustrates an embodiment of an exemplary network implementation4300. In FIG. 43, NEs 4302 a-4302 n (collectively 4302) contain an OSIprotocol stack as defined by 3102 a-3114 a, 3102 b-3114 b, 3102 c-3114c, and 3102 n-3114 n. Additional NEs may exist in the network havingsimilar structures as defined by NEs 4302.

Network Layer through Application Layer, 3106 b-3114 b and 3106 c-3114c, may not exist in some types of NEs, such as Ethernet Switches. Inother cases, such as with routers, some additional layers above the DataLink Layer 3104 b and 3104 c may exist. It is the existence of theselayers above the Data Link Layer 3104 b and 3104 c where someembodiments of the present invention take place. The Physical Layer 3102a-3102 n generally exists in each NE and is included in the embodimentsof the present invention.

Bi-directional MEs 4304 and 4306 exist and operate in the OSI referencemodel Data Link Layer 3104 a-3104 n. An end-to-end user communicationpath may be defined by each of 3114 a, 3112 a, 3110 a, 3108 a, 3106 a,3104 a, 3102 a, 3102 b, 3104 b, 3104 b, 3102 b, 3102 c, 3104 c, 3104 c,3102 c, 3102 n, 3104 n, 3106 n, 3108 n, 3110 n, 3112 n, and 3114 n.Within NE 4302 b and 4302 c, information flows up from the PhysicalLayer to the Data Link Layer and then back down from the Data Link Layerto the Physical Layer as it is processed by each NE. The informationflow can either be full duplex (bi-directional paths operatingindependently from each other at the same time) or simplex (operating inone direction at a time, but in both directions) or uni-directional(operating in one direction only).

FIG. 44 illustrates another embodiment of an exemplary networkimplementation 4400. As described above, bi-directional MEs 4304 and4306 exist and operate in the OSI reference model Data Link Layer 3104a-3104 n. Within NEs 4302 a-4302 n, performance information can beextracted from PIP packets. Once extracted, this information can be sentvia communication pathways 4402-4408 to the Physical Layer 3102 a-3102 nprotocols where the operation of these Physical Layer protocols can bemodified to react to real-time information provided via the performanceinformation.

FIG. 45 illustrates an embodiment of a wireline Digital Subscriber Loopnetwork 4500, including an Ethernet Router/Switch 4514, Ethernet DataStream 4510, Ethernet Management Stream 4512, Network/EthernetManagement System 4508, and Internet Service Providers (ISPs #1 and #2)4516 a and 4516 b. One example of how performance monitoring ofnon-Ethernet segments may be utilized is in monitoring of broadbandaccess Digital Subscriber Loop (xDSL) connections. The term, “xDSL”generally means DSL technologies, such as ADSL, ADSL2, ADSL2+, VDSL,VDSL2, etc. By extracting and inserting relevant DSL performancestatistics into the PIP packet, a single management system may havevisibility to the end-to-end performance of a customer's connection.This insertion could include appending this performance information tothe end of the payload of the PIP packet as previously described. Thisappending may occur at the DSL DSLAM 4504 (assuming the NetworkConnection is Ethernet). The PIP packet containing the DSL networkperformance information 4506 may then be made available to the OperatorsNetwork Management System 4508.

The additional DSL network performance information improves repairresolution time, as network problems at a subscriber may be quicklyidentified. The DSL network performance information allows fullmonitoring and troubleshooting through a single network managementsystem. In addition, dynamic configuration changes based on networkperformance may be made by the Network Management System 4508 tooptimize circuit performance. For instance, if a DSL circuit 4502suddenly experienced a peak or spike in impulse noise on thenon-Ethernet segment, appropriate diagnostic information may be insertedinto the Ethernet management stream and adjustments may be temporarilymade by the network management system 4508 to the DSL Signal-to-Noiseratio to compensate for the interference and to ensure line stability.After a given timeframe or due to improving changes in capturedperformance data, the line could be re-provisioned by the managementsystem to improve overall performance. This example is one of dozens ofpossible configuration changes that may be made in real-time to optimizeDSL circuit performance.

Other exemplary network performance information parameters that may becaptured and inserted into the management stream include: near-endfailures, far-end failures, last state transmitted (downstream andupstream), actual signal-to-noise ratio, maximum attainable data rate,actual power spectrum density, actual aggregate transmit power, xDSLprofile, xDSL limit PSD mask and band-plan, xDSL Power Spectral Densitymask, estimated upstream power back-off electrical loop length, trelliscode use, actual cyclic extension, band number, line attenuation perband, signal attenuation per band, signal-to-noise ratio margin perband, actual data rate (downstream and upstream), previous data rate(downstream and upstream), actual interleave delay (downstream andupstream), actual impulse noise protection, impulse noise protectionreport, actual size of Reed-Solomon codeword, actual number ofReed-Solomon redundancy bytes, actual number of bits per second, actualinterleaving depth, actual interleaving block depth, actual latencypath, interval number, interval status (valid and complete; invalid orincomplete), forward error correction seconds, errored seconds—line,severely errored seconds—line, loss of signal seconds—line, unavailableseconds—line, full initializations, failed full initializations, shortinitializations, failed short initializations, sync mode, or othercapabilities identified in xDSL (e.g., ADSL1, ADSL2, ADSL2+, VDSL2,etc.).

Specifically, the T1.413 Standard defines methods to dynamically adaptthe DSL transport stream, the subject matter of which is herebyincorporated by reference. These dynamic adaptations are described inthe T1.413 standard under the sub-section “On-line adaptation andreconfiguration using the Overhead Control Channel (AOC)”. In thissub-section the standard defines that the AOC data is carried asoverhead bytes in the DSL framing structure. The actual multiplexing ofthese overhead bytes into the DSL framing structure depends on theframing structure used (i.e., full overhead or reduced overhead) and onthe allocation of any bearer channel to the fast or interleaved databuffer.

The type and length of an AOC message (except for the acknowledgemessages) are identified by a byte-length header. In particular, the AOCchannel sends an all binary zeros “00000000” AOC stuffing pattern in theIdle State, and a valid AOC message always begins with a non-zero byte.

The T1.413 Standard further defines “On-line adaptation—Bit swapping.”Bit swapping enables a DSL system to change the number of bits assignedto a sub-carrier or change the transmit energy of a sub-carrier withoutinterrupting data flow. An ATU (DSL Termination Unit) may initiate a bitswap. The swapping procedures in the upstream and downstream channelsmay be independent and may be performed simultaneously. For the bit swapprotocol, the “receiver” is the ATU that is receiving the data; ittransmits a bit swap (extended or simple) request message and receivesthe bit swap acknowledge message. The “transmitter” is the ATU that istransmitting the data. It receives a bit swap request (extended orsimple) message and transmits the bit swap acknowledge message.

Bit Swap Request Commands.

DSL information or other network performance information may be used todynamically alter some performance parameters of the Physical Link, suchas transmit power. A sub-process may be established in the DSL unitwhich would monitor PIP packets and then issue the proper AOC BitSwapping commands (see FIG. 46) to affect the necessary performancerequirements.

FIG. 47 illustrates an embodiment of an exemplary wireless network 4700that operates in accordance with the principles of the presentinvention. Many types of wireless communications may benefit from theinsertion of performance data into a PIP packet. By extracting andinserting relevant wireless segment performance information into the PIPpacket, link performance problems may be detected, and traffic couldthen be rerouted via a centralized management system. Likewise, if themanagement system determined that a user could achieve greater overallperformance by routing traffic in a different manner, that users trafficmay be diverted from the current path, even if that path may have thestrongest wireless signal. For example, a wireless device 4702communicates with a wireless access point 4704 via a wireless path 4706.In one example, the wireless path 4706 between the wireless device 4702and the wireless access point 4704 has strong signal strength. As thequantity of users on this link increases and performance over thewireless path 4706 degrades, network performance information containedin PIP packets collected by the Network management System (NMS) 4712 maytrigger a wireless network management system to send instructions to thewireless device 4702 to redirect traffic to wireless access point 4708.by redirecting wireless traffic to the wireless access point 4708;performance of the wireless access point 4704 may improve and, inresponse, traffic may be redirected from the wireless access point 4708to the wireless access point 4704 to accomplish load balancing.

The network performance information indicative of a problem at awireless access point may be appended to the end of the payload of PIPpackets. This appending may happen at the wireless routers 4704 and4708. The PIP packets containing the wireless performance information4710 would be available to the Network management System, which, inturn, may instruct the wireless device 4702 and the wireless accesspoint 4704 to disconnect and reconnect the wireless device 4702 viawireless access point 4708, in one embodiment. This disconnection canoverride other wireless connectivity parameters, such as signalstrength. Alternatively, the NMS 4712 could instruct the wirelessdevices to switch to a different channel. Other similar variations arealso possible to re-route or re-channel the wireless device 4702.

One example of re-routing may be as follows. If the NMS system 4712determines that connectivity to ISP #2 4516 b may provide betterperformance than ISP #1 4516 a, traffic would be dynamically rerouted toISP #2 4516 b based on rules, thresholds, etc., that the NMS 4712 couldapply to network performance information collected at either or both ofthe ISPs 4516 a and 4516 b. Other exemplary variables include: wirelesschannel, encryption level, and connectivity mode (802.11a, 802.11b,802.11g, 802.11n, WiMax, etc.). The NMS system 4712 may constantlymonitor the network performance information data flows throughout thewireless network 4700 and evaluates traffic and paths based on thenetwork performance information contained in the PIP packets. Inresponse to determining that one or more node segments areunderperforming, calls may be rerouted. The NMS system 4712 may evaluatethe line state at and between each the connection points within thewireless network 4700. In one embodiment, the NMS system 4712 mayevaluate the core network, including trunk segments, in addition toevaluating the wireless access points 4704 and 4708. The NMS system 4712may retest connections on a periodic basis, such as every 10 seconds or10 milliseconds, for example.

Although shown as a portable computer, the wireless devices 4702 mayalternatively be a phone, PDA, and/or any wireless device that may usethe wireless network 4700 to communicate. Although two wireless accesspoints 4704 and 4708 are shown, any number of wireless access points maybe used with the NMS system 4712 and wireless network 4700.

Further, any number of wireless networks 4700 may be used for evaluatingand routing wireless calls. For example, NMS system 4702 determines thata particular wireless network is having difficulty carrying calls due tocongestion or other technical problem, then the NMS system 4702 mayswitch or route its calls to another wireless network. In addition, ifthe NMS system 4702 determines that a particular signal strength from awireless device to a wireless access point is weak or becomes weaker dueto any number of factors, including due to the user increasing distancebetween himself and the access node, then the NMS system 4702 may changeone or more communications parameters, including encoding, modulation,frequency, and the like to improve or increase the signal strengthbetween the user and the access node. Changing communication parameters)could be done automatically or manually via a button on the userswireless device that initiates a request for determining why the signalstrength is degrading or decreasing, and, in accordance with theprinciples disclosed herein, the information derived from the PIP packetmay be used to determine these or other solutions to improving thesignal or increasing the signal strength. In yet another embodiment, theNMS system 4702 may troubleshoot the networks automatically without anyuser initiation on a periodic basis to report back on the status ofthese wireless connections.

Network Layer Example.

The Network Layer 3106 may determine how data is transferred betweennetwork devices, route packets according to unique network deviceaddresses, and provide flow and congestion control to prevent networkresource depletion. For purposes of this invention, routing protocolsare defined as the protocols used in the implementation of routingalgorithms to facilitate the exchange of routing information betweennetworks. This exchange of routing information allows NEs defined asrouters on the Network Layer 3106 to build routing tables on a dynamicbasis.

In one embodiment, the principles of the present invention provides forthe injection of dynamic link state information obtained from PIPpackets into routing algorithms. In FIG. 48, the Data Link Layers 3104a-3104 n may make network performance information in the PIP packetavailable to one or more of the Network Layers 3106 a-3106 n. Thenetwork performance information may include data associated withcommunications of data packets including real-time content. Data flowover the Data Link Layer may include network performance informationderived from either of the MEs 4304 and 4306 to Network Layer routingprotocols and routing protocol metrics. This is shown as data flows4802-4808.

In essence, real-time network performance information, such as linkfailure, link degradation, MEF TRAN failure, Label Switch Path (LSP)ping, trace-route, Virtual Circuit Connection Verification,Bi-Directional Forward Detection, MPLS Fast Reroute, and other similarcapabilities may be dynamically inserted into Link State routingprotocols, thereby forcing recalculation of the route tables,calculation of optimal route paths, and potential Link StateAdvertisement (LSA) re-advertisement. LSAs are processes to updateneighbor nodes in the event of a Link State change. The LSA processtypically creates a short message (i.e., the link-state advertisement)which: 1) identifies the node which originates the LSA; 2) identifiesall the other nodes to which it is directly connected; or 3) includes asequence number, which increases every time the source node makes up anew version of the message. This message is then communicated throughoutthe network. In one embodiment, the link-state message is communicatedto all other network nodes on the network. Typically, each node in thenetwork is responsible for storing the sequence number of the lastlink-state message which it received from other nodes. Once the LSAprocess completes, each node uses this information in calculations foran optimal routing path to other nodes on the network. This informationmay be included as routing metric information by Network Layer routingalgorithms.

Currently, many common implementations of routing protocols, such asOSPF, establish a link cost to be proportional to the inverse of thelink bandwidth. A use of the ME information may be to modify the linkcost to represent a larger value; hence, the cost rises and the use ofthe link is less likely. Once congestion clears in the ME, the link costcould be reestablished to reflect the normal setting.

Real-time dynamic link information can significantly enhance link statepacket routing protocols. The dynamic link state injection into packetrouting protocols may offer the capability to: 1) sample the quality ofthe physical connection and proactively react to failing conditions, 2)assess changing traffic flows over the connection at regular intervalsproviding per flow traffic rerouting to accommodate optimal performance,3) proactively react to degradation conditions affecting the circuitsuch as creating alternative paths and rerouting traffic prior tocircuit failure, 4) load balancing traffic flows over multiple circuitsto accommodate circuits that are operating at less than optimalconditions, 5) improve route re-convergence times, 6) eliminate someroute ‘flapping’ conditions and 7) other similar types of routeenhancing capabilities.

FIG. 49 illustrates an embodiment 4900 of the injection of dynamic linkstate information into Transport Layer protocols and algorithms. In thisTransport Layer embodiment, such information flows into and out of MEs4304 and 4306 as data flows 4902-4908. In one embodiment, thisinformation flows via PIP packets.

FIG. 50 illustrates an embodiment 5000 of a TCP packet in accordancewith the principles of the present invention. An example of the dataflows into the Transport Layer 3108 would be a congestion notificationvia an PIP packet into the Transport Layer 3108 a-3108 n. TCP is acommon protocol that operates in the Transport Layer. In TCP, a slidingwindow is a variable flow control mechanism to manage the efficiency oftransmission on the network. The TCP Sliding window allows a sender totransmit a specified number of data units before an acknowledgement isreceived or before a specified event occurs, such as a timer expires.The TCP window function also has a tributary effect on the quantity ofpackets that can be transmitted during a time window given that the TCPprotocol requires a far-end acknowledgement that a window-size ofpackets was received prior to transmission of the next packet. When thephysical distance between end-points becomes large, the acknowledgementtime becomes a significant contributing factor in the reduction ofeffective capacity of a call path. The TCP window can be set to a largersize to increase the transmit versus wait for the acknowledgementwindow. However, this setting has potential to cause congestion on localLAN segments given the Ethernet collision domain. In this case, theembodiment uses either an ad hoc PIP packet or existing PIP packet ifthe user is using a static VPN or Point of Presence protocols, such asSLIP or PPP. In this example the MEP and/or Protocol stack contains thePIP PM information, which may include a round trip delay measure. Witheach TCP session, the TCP protocol can automatically check the PIP PMinformation and dynamically adjust the TCP window size to meet theline-state conditions. This closed-loop system effectively automates theTCP window setting to the optimal setting for obtaining throughputperformance. It should be understood that the same delay information canbe used to alter the TCP time-out windows. The ME performanceinformation indicative of congestion may be used to directly adjust(reduce or increase) the TCP Sliding Window 5002, which in turn, reducesthe quantity of packets defined by the window that could requireretransmission due to lost packets.

In a packet network, an intermediate node that identifies networkdegradation, delay, congestion, and the like, may capture PIP packetinformation, which may include event data, and propagate the informationto other NEs using PIP packets. At the same time, the intermediate nodemay also inject this information into the ACK packet flowing from thereceiver to the sender. After getting this information, the sender maychange the window size and follow up with other appropriate action.

Those data packets that may be flowing in the reverse direction, thewindow size modification information can also be passed to the receiverto take appropriate action. In one aspect, the information contained inPIP packet may be stored externally to the OSI stack or injected into aTransport Layer 3108 device and be stored as a line-state to effect thechange of the TCP window size. The NEs may have the lower TransportLayer 3108 protocols, and if not, then another downstream NE that hasthe ability may be responsible for this action.

At each end of the TCP connection, buffers may be used to manage thedata flow. This management may be in a form of flow control and uses theTCP Sliding Window 5002 to perform this flow control. In the TCP SlidingWindow function, a window is defined as the maximum number ofunacknowledged bytes that are allowed in any one transmission sequence.The receiver of a packet flow specifies the current receive TCP SlidingWindow 5002 in every packet sent to the originator. The sender may sendup to the amount of specified in the TCP Sliding Window 5002 before ithas to wait for an update on the TCP Sliding Window 5002 (from thereceiver). It should be understood that the TCP function may be modifiedto send a PIP parameter modified window and time-out settings to thefar-end during the initiation of the TCP session itself and/or query PIPinformation stores contained on the end-points

That the sender network node buffers its own sent data until it receivesacknowledgements (ACKs) for the sent data. The TCP Sliding Window 5002size is typically determined by whatever is the smallest between theReceive Window and the sender's buffer. The TCP Sliding Window 5002field indicates the range of acceptable sequence numbers, beyond thelast segment, that has been received successfully. This value is theallowed number of octets that the sender of the ACK is willing to acceptbefore an acknowledgement. As the TCP process performs the transmissionof a segment of data, it places a copy of the data in a retransmissionqueue and starts a timer. If an ACK is not received for that segment, ora part of that segment, before the timer runs out, then the segment, orthe part of the segment that was not acknowledged, is retransmitted.This embodiment directly modifies values contained in the TCP SlidingWindow 5002 or other portion of TCP Packet 5000 as it traverses througha NE with Layer 4 capabilities.

Within TCP packet 5000, individual code bits flags are identified asfields “U,” “A,” “P,” “R,” “S,” and “F” are used to indicate the natureof the header in relationship to the protocol conversation. For example,such fields include U—Urgent Pointer (URG) 5004, A—Acknowledgement (ACK)5006, and P—Push function 5008. Push function 5008 causes the TCP senderto push all unsent data to the receiver rather than sending segmentswhen it gets around to them, (e.g., when the buffer is full). Additionalfields typically found in a TCP packet 5000 include: R—Reset theconnection (RST) 5010, S—Synchronize sequence numbers (SYN) 5012, andF—End of data (FIN) 5014.

Also within TCP packet 5000 are three other fields that may be directlymodified to assist in the shaping of the traffic flow between sender andreceiver. The first field is the Window field that indicates the rangeof acceptable sequence numbers beyond the last segment that has beenreceived successfully. A value of the window field represents theallowed number of octets that the sender of the ACK is willing to acceptbefore an acknowledgment. The second field is the Urgent Pointer 5016that shows the end of the urgent data so that interrupted data streamscan continue. When the URG bit 5016 is set, the data is given priorityover other data streams. The last field is the Options 5018 that maycontain a TCP Maximum Segment Size (MSS) and is sometimes called MaximumWindow Size or Send Maximum Segment Size (SMSS).

In one embodiment, the NE 4302 b-4302 c supports a protocol stackthrough the OSI Transport Layer 3108. In addition, a NE 4302 b-4302 csupporting this embodiment may contain a set of embedded programminginstructions that would react to the ME performance information,establish which fields in the TCP packet would be modified, modify thefield values, and send the packet out of the egress interface.Modifications could be made to packets traversing in any direction(sender to receiver, receiver to sender, or both).

In another embodiment, the OAM information obtained from the Data LinkLayer 3102 and contained in a PIP packet is used in other types ofprotocols, such as the User Datagram Protocol (UDP). Since UDP does notuse a window or acknowledge packet receipts like TCP/IP, there is nocontrol on the sending rate. Nevertheless, the rate may be controlled bysetting limits on the maximum bandwidth allowed between sites used inother applications and protocols, such as File Transfer Protocol,Database Storage, and Voice over Internet Protocol (VOIP). Once thenetwork problem is detected, the Data Link Layer 3102 information,derived from the PIP packet, may be injected into Transport Layer 3108to make the source control the bandwidth. The fault identificationprocess may be refreshed at certain intervals so as to get the currentstatus. Once the fault goes away, this indication may be injected intothe Transport Layer 3108 data so that the appropriate NE can take thenecessary action and the original transmission rate can resume.

In another embodiment, the information contained in the PIP packets maybe communicated to the protocol stacks contained with Network Layer 3106devices, such as routers, to convey that there may be the potential forcollapse due to congestion. This congestion avoidance may be achieved bypacket queuing and/or packet dropping techniques to slow down excessiveUDP traffic. Further, Datagram Congestion Control Protocol (DCCP) may beused to add end host TCP-related congestion control behavior tohigh-rate UDP streams, such as streaming media.

FIGS. 51-53 illustrate corresponding exemplary embodiments 5100-5300 ofdata flows to other layers in the OSI reference model.

Further to the discussion above to FIG. 50, FIG. 54 illustrates anembodiment of an exemplary method for adjusting TCP window size when thePIP OAM performance and utilization information indicates a fault in themiddle of the network. In this embodiment, a fault is detected in themiddle of the network and the fault information is injected into theAcknowledgement (ACK) packet before being sent to the packet-sendingnetwork device (sender). Upon receiving the information, the sender maychange the window size. Typically, a sender starts with an initial TCPwindow size.

In step 5402, the capability of the network element to process layer 4information is determined. In step 5404, the PIP ME performance andutilization information is captured and sent to the layer 4 embeddedprogramming process in step 5408. In these steps, a QoS or trafficproblem may be identified at a Data Link Layer 3104 node. Also, in step5408, the sender may receive the ACK packet from the receiver. The faultinformation may be injected from Data Link Layer 3104 to Transport Layer3108. This may include embedding this information in the TCP ACK packet.In step 5408, the nearest NE with Transport Layer 3108 may also beidentified. In step 5416, the sender receives the ACK with faultinformation. In step 5410, the embedded program determines if the PIPperformance information indicates a performance degradation. If so, thenfor each TCP packet received at the ingress of the NE at step 5412, adetermination is made at step 5414 to determine if the TCP state isestablished. If in an opening state or closing state, then the NE may donothing. If the TCP state is in an established state, then the currentwindow size is viewed and determined if it is at zero (0). If it is atzero, then the TCP inherent flow control mechanisms may have alreadytaken care of the congestion problem. If the window size is non-zerothen the TCP window size 5418 is reduced and the packet is sent out theegress 5420. In step 5418, the sender may make additional appropriatechanges in the TCP window size, such as increasing the window size.

In addition to the embodiments above, FIG. 55 illustrates an embodimentof an exemplary system 5500 and method for shaping network traffic(“network traffic shaping system”) that includes using an Ethernet FirstMile OAM Packet, PIP packet, or other suitable packet to dynamicallychange traffic shaping to minimize bursting and packet loss on a packetswitched network. FIG. 55 illustrates a typical packet network 5502including a broadband remote access server (BRAS) 5504 and a DSLAM 5506interconnected across the packet network 5502 in which the networktraffic shaping method operates. Typically, the packet network 5502operates in Data Link Layer 3102 (FIG. 31) of the OSI reference modeland typically includes data link communication devices, or data linklayer devices, such as bridges and switches. Generally, bridges andswitches extend the effective length of a LAN by permitting theattachment of distant stations. It should be understood that the virtualpacket path between the BRAS, DLSAM, DSLAM modem, and User CPE devicesmay traverse any type of packet network or transport schema.

The packet network 5502 may support Data Link Layer 3102 or NetworkLayer 3106 network facilitating Data Link Layer 3102 tunnels or anyother packet network that supports Data Link Layer 3102 tunneling, suchas Asynchronous Transfer Mode (“ATM”). In another embodiment of thepresent advanced fail-over method, LAN switches are used to interconnectmultiple LANs.

Some common switching technologies used with the present network trafficshaping system 5500 include store-and-forward switching and cut-throughswitching. Typically, store-and-forward switching waits for an entireframe, or encapsulated packet(s), to be received before forwarding. Incut-through switching, the switch begins forwarding the frame whenenough of the frame is received to make a forwarding decision. It shouldbe understood that the BRAS function is a universal edge shapingfunction that can be distributed throughout the network. The shapingfunction is normally statically set to a circuit performance level,which shapes and discards traffic to meet a specific performanceparameter regardless of what is occurring downstream in the network.

In addition, the network traffic shaping system 5500 may operate withcommon bridges including transparent bridging as found in Ethernetenvironments, source-route bridging as commonly found in Token Ringenvironments, and source-route transparent bridging as commonly found inmixed Ethernet/Token Ring environments.

The BRAS 5504 in the instant shaping example, is typically the gatewaybetween the Internet and DLSAMs in the network accessed by DSLAMcustomers. The BRAS 5504 or shaping entity may contain a MEP. The MEPmay track performance or it may obtain network performance informationof the customer and node or trunk levels from somewhere within the OSIstack. In one embodiment, the BRAS 5504 can use the transmission pathstate information by detecting transmission Frame Loss via PMinformation contained within the PIP packets. The PIP session(s) can beboth trunk level to a network node, such as a DSLAM or to a customerlevel. In both cases the trunk or line state PM engine detects thattransmitted packets passing beyond the shapers are being dropped in thepacket network 5502 or aggregation device going to the DSLAM 5506, theDSL modems, or customer CPE. In one embodiment, the PIP PM statecommunicates transmission loss to the traffic shaping mechanisms, areaction to the transmission path congestion or degradation and canfurther limit the transmission rates to alleviate the congestion bymodifying (lowering) the shaper windows. In one example, transmissionloss causes the network to slowly begin dropping these packets in thepacket network, and the shapers react by performing one or more shapingor scheduling functions through the BRAS 5504 to stop the packets frombeing dropped in the packet network. It is understood as the packettransmission rate increases or frame loss rate decreases that theshaping and scheduling functions should gracefully increase thethroughput window, thereby returning the circuit state to its normalcondition. As a result of these functions, the packets are droppedbefore entering the packet network, thus not tying up bandwidth withpackets that are ultimately dropped within the packet network 5502.

In one embodiment, the BRAS 5504 is used to shape DSL traffic of eachuser 5510 by using end-to-end signaling outside of TCP flow control toadjust the bursting to eliminate congestion. In one embodiment, the IPprotocol flows anticipate lost packets and enables dropping thesepackets prior to them being dropped in an internal network, therebyeliminating congestion control and reducing network burst traffic, whichincreases the amount of capacity required by the network.

In one aspect, a CPE, such as DSLAM 5506, receives network performanceinformation, via PIP packets, that a percentage of the packets that weresent by the BRAS 5504 did not get delivered to the DSLAM 5506. In oneembodiment, the modified Y.1731 protocol PM stack is used to transmitthe receive PM information performance data from the DSLAM 5506 to theBRAS 5504. The protocol performs algorithms to determine the number ofdropped packets not received by the DSLAM 5506. Thus, the DSLAM 5506contains a Y.1731 stack that correlates this information. There arethree general ways in which this data is transmitted to the BRAS 5504, atrigger, a push, and a pull, as commonly known to those skilled in theart. It should be understood that to accommodate the “forward” shapingat the edge of the network a PIP PM data must be returned to the shapingentity from the far end. The relay of the PM information from MEP to MEPis understood to occur at the node trunk level, node port level,Customer NID device, and Customer CPE to obtain each subsequent level ofshaping.

The BRAS 5504 performs one or more of several functions to “self-heal”the packet network 5502. For example, the BRAS 5504 can decrease thedata transfer rate to each user 5510 from the Internet 5508 to thepacket network 5502, thus causing the packets to be dropped prior toentering the packet network 5502 and avoiding the packets later beingdropped in the packet network. The BRAS 5504 provides for real-timedynamic network traffic shaping based on session flow performance of thefar-end based on the performance data included in a received packet.Thus, if a CPE, such as the DSLAM 5506, is communicating through the PIPpackets that packets are being dropped in the packet network 5502, theBRAS 5504 relieves the congestion on the packet network by decreasing,via filter 5512, session flow performance of the users 5510 at the BRAS5504 prior to the session flow entering the packet network 5502.

The foregoing exemplary BRAS operation reduces provisioning complexityby adding a “self-healing” traffic shaping mechanism to the networkInternet service point. The present network traffic shaping system alsoenables a “plug-and-play” traffic adjustment scheme in that a user maychange the network and the BRAS and/or other network elements willdynamically learn the available transport capacity and adjustaccordingly without being manually configured.

In one embodiment, the network traffic shaping system 5500 includes anodal mass calling shaping congestion control function for shaping therate of data traffic through a network based on PIP and PM packetinformation. The network traffic shaping system may enforce a queue ortraffic shaping for an entire access node 5516 or group of traffic in anaccess node 5516. Using the same principles as discussed above, alltraffic from an access node may be placed into a virtual packet circuit(IP, Mac Address, VLAN, LSP, etc.) and built into a scheduler so theoverall traffic 5520 may be shaped. Without knowing what is in themiddle of the packet network 5502, the traffic shaping system 5500 cantrack the packets received at the access node 5516 and use that data atthe BRAS 5504 to rate shape the entire group 5522 of customers 5510.This rate shaping preserves the cross-utilization of bandwidth betweenthe customers 5510 on access node 5524 and minimizes the packet loss dueto bursting and mass calling events that might occur in the network.

In one aspect, the traffic shaping function performed by the networktraffic shaping system 5500 may be based on cutting the non-real timedata flows versus the real-time data flows. In another aspect, thetraffic shaping function performed by the network traffic shaping system5500 may be based on QoS requirements and obligations to determine whichdata flows to drop and which data flows to keep transmitting. In yetanother aspect, the traffic shaping function performed by the networktraffic shaping system 5500 can also look at the (i) DIFFSERV protocolmarking, (ii) recipient's IP address, and (iii) trunk from which data isbeing received to determine what flows to shape or drop altogether.

Generally, the present network traffic shaping system 5500 mapseverything to a Media Access Control (MAC) address, an Ethernet virtualcircuit, a PPPOE tunnel, a PPPOA tunnel or other similar structures.These locations are considered the egress points. Some or all of thefollowing criteria may be used to determine how data is to be rateshaped. As discussed above, the present network traffic shaping system5500 can determine what port the data came from. For example, adetermination can be made as to whether the port is an Internet dataport or a VOIP data port. In one aspect, the present network trafficshaping system 5500 may determine to drop the traffic coming from oneport or shape one port to another port. In a second way, the presentnetwork traffic shaping system 5500 may have two virtual circuits goingdown to the CPE or DSLAM 5506, so it may shape one of them versus theother. In a third way, the present network traffic shaping system 5500may have a large bandwidth going down to the CPE or DSLAM 5506, but itmay use a priority bit marking in the packets to choose which traffic toshape or drop. In a fourth way, the present network traffic shapingsystem 5500 may determine the type of packet that is sent through thepacket network 5502. For example, if the packet is a real-time VOIPpacket and another packet is a FTP packet, then it may decide to dropthe FTP packet and transmit the VOIP packet. In yet another example, thepresent network traffic shaping system 5500 may determine that aparticular user 5510 has a multiple of IP addresses for a particularcustomer 5510 and decide to not transmit for a period of time on one ofthose IP addresses.

FIG. 56 illustrates an embodiment of an exemplary user interface for thepresent network traffic shaping system 5500. The Normal Rate Cap field5602 contains values relating normal rate capacity of a particularsession or data flow for either a particular user 5510 (FIG. 55) or anode 5516. The Minimal Cap Rate field 5604 contains values for theminimal rate capacity for a particular session or data flow. The BW StepDown Method field 5606 and the BW Step Up Method field 5608 eachcontains values and related algorithms relating to the amount of thestep-wise reduction or step-wise increase performed by the presentnetwork traffic shaping system 5500 when reducing the bandwidth toeither the user 5510 or the node 5516. The Threshold Loss to Step DownBW field 5610 and Threshold Condition to Step Up BW field 5612 eachcontains values and related algorithms for initiating the step-wisereduction or step-wise increase of the Step Down Method field 5606 andthe BW Step Up Method field 5608.

FIG. 57 illustrates an embodiment 5700 of an exemplary method forshaping the rate of data traffic through a network based on informationfrom the PIP packet of the present network traffic shaping system 5500(FIG. 55). In step 5702, the data flow is initiated. In step 5704, thePIP packet data is collected by a CPE or DSLAM 5506 as discussed herein.In step 5706, the PIP packet data is transmitted from the CPE or DSLAM5506 to the BRAS 5504. In step 5708, the present network traffic shapingsystem 5500 queries whether the Y.1731 type packet loss threshold hasbeen exceeded as described above. If it has, then the present networktraffic shaping system 5500 further queries whether the minimumbandwidth threshold has been reached in step 5710. If it has, then thepresent network traffic shaping system 5500 incrementally reduces thecustomer or access node shaping window at step 5712.

In step 5708, if the Y.1731 type packet loss threshold has not beenexceeded, then the present network traffic shaping system 5500 querieswhether the maximum threshold has been reached in step 5714. If themaximum threshold has been reached, then the present network trafficshaping system 5500 continues at step 5706, where PIP packet data iscollected from the CPE or DSLAM 5506 at the BRAS. In step 5716, thepresent network traffic shaping system 5500 queries whether thebandwidth set-up threshold is met. If the set-up threshold has been met,then the network traffic shaping system 5500 incrementally increasesbandwidth shaping at step 5718, as described above.

In another embodiment, an Application-Specific Integrated Circuits(ASIC) directs packet data flow and PIP packets based on their real-timeand non-real-time content. FIG. 58 illustrates an embodiment 5800 of anexemplary Data Link Layer device 5804 and an ASIC device 5802 that isassociated with an incoming network interface for communicating to anoutgoing network interface. The ASIC 5802 may be capable ofcommunicating switched data to an outgoing network interface alsoassociated with the Data Link Layer device 5804. The ASIC 5802 isdesigned to separate or buffer particular data flows, such as data flowsof data packets including real-time and non-real-time content. The dataflows may be performed over a network interface card (NIC) operating ina computer Ethernet port or pluggable fiber/electrical modules.

The Data Link Layer device 5804 includes one or more incoming networkinterface or module 5812 a-5812 n (collectively 5812) and one or moreoutgoing network interface or module 5814 a-5814 n (collectively 5814).The network interfaces 5812 and 5814 may be capable of handling packetbased and other suitable digital signals associated with voice, video,and data transmissions of a packet switched network. In addition, theData Link Layer device 5804 may also include switching unit controllers,processors, memory, and buses interconnecting them, as known in the art.The network interfaces 5812 and 5814 may also be capable ofcommunicating with other network interfaces, single or multiple, such asT1 interfaces, E1 interfaces, Integrated Services Digital Network (ISDN)interfaces, SS7 interfaces, Optical Carrier level-3 (OC-3) interfaces,other optical interfaces, any other types of interfaces, or combinationsof these interfaces.

The ASIC 5802 may also include one or more incoming lines 5816 a and5816 b (collectively 5816) and one or more outgoing lines 5818 a and5818 b (collectively 5818) that may in communication with other devicesof the packet network. The ASIC 5802 may also be connected directly tothese lines 5816 and 5818 or may be connected via a bus of a suitabletype, such as control, synchronization, power, isolation, serial, andthe like. In one embodiment, the ASIC 5802 may receive uni-directional,bi-directional, or other serial data streams incoming from the incomingnetwork module 5812. Moving in the opposite direction, the ASCI 5802 maytransmit separated or stored real-time data flows and non-real-time dataflows through outgoing network modules 3212. In one embodiment, ASIC5802 may also contain a processor(s) 5806 memory 5808, such as ROM, RAM,EEPROM, Flash, and the like, and coded logic or software 5810 forperforming the operations described herein. Memory 5808 may storeregisters, such as sampling registers and static registers based on thetype of data flow through the ASIC 5802.

In one embodiment, the one of the two outgoing data path lines 5818 aand 5818 b is used for transmitting real-time data flow and the otherfor non-real-time data flow. Once the two data flows are separated atthe ASIC, they can each be transmitted to their respective lines 5818 aor 5818 b for measuring in accordance with the principles of the presentinvention.

In one embodiment, the ASIC 5802 creates “sampling” shift registers withQoS or other mapping/replication functions. In one aspect, the samplingshift registers measure the “buckets” or total volume or amount ofeither real-time data content or non-real-time content through a DataLink Layer device 5804 or other network device. The ASIC 5802 maymeasure the total volume, as opposed to bit transfer rate, of aparticular data flow that includes either real-time or non-real-timecontent. For example, the ASIC 5802 may measure an interval of trafficfrom one of these data flows and then measure another interval oftraffic in a “round-robin” manner. A static register allocates a bucketper stream, bucket per flow, bucket per logical connection, bucket perport, and/or bucket per device. Further, the ASIC 5802 may contain ascheduler modification to provide actual scheduler performanceinformation on what flows are being served at what rates. Also, it mayinclude a “settable counter trigger” that counts when a packet has aspecific TOS, QoS, or other marking. Externally, the ASIC 5802 maycreate a “line state” dongle or inline probe that measures, via anymethod, and generates the PIP packet information in both directions forreal-time and total bandwidth with other peakedness measures. The term“peakedness” means within-the-hour or moment-to-moment variations intraffic.

Further, the coded logic or software 5810 of the ASIC 5802 may determinepackets that contain real-time data content from those that containnon-real-time data content based on the port or device that transmittedthe packet to the ASIC 5802, the payload of the packet, the p-bit of thepacket, header information of the packet, or by any other means commonlyknown to those skilled in the art and/or described herein. Further, theASIC 5802 may store the TOS, QoS, or other service information relatedto a particular customer or user associated with the packet or flow ofpackets.

In addition, the ASIC 5802 may characterize the “peakedness” or burst inreal-time. The real-time data may be characterized as a “state,” thusmaking it useful for mathematical calculations and algorithms todetermine the amount of real-time data content transmitted.

FIG. 59 illustrates an embodiment of an exemplary method 5900 fordetermining the amount of real-time data flow and non-real-time dataflow with an ASIC 5802. In step 5902, the data is transmitted to theASIC 5802 in any manner commonly known to those skilled in the artand/or described herein. In step 5904, the ASIC 5902 determines thepackets that contain real-time content and packets that containnon-real-time content. In step 5906, the real-time content packets areseparated from the non-real-time data packets based on features of thepackets or sender information associated with the packets as describedherein. In step 5908, the volume of real-time content is measured inbuckets or other means. In step 5910, the volume measurements ofreal-time content may be transmitted to other devices or systems for usein adjusting these devices and systems to optimize the real-time contentflow through a network in step 5912. In addition, billing considerationscan be made based on the total volume of real-time data content measuredat the ASIC 5802.

FIG. 60 illustrates an embodiment 6000 of a method for using informationcontained in PIP packets to control packet traffic flow with UDP. Instep 6002, a QoS or traffic problem is identified at a node in the DataLink Layer 3104. In step 6004, a check on NEs Data Link Layer 3104capabilities is performed. In step 6006, the fault identificationprocess refreshes at certain intervals. In step 6008, the NE may controlthe flow of packet transmission using UDP and buffer size. In step 6010,fault information is injected from the Data Link Layer 3104 into theTransport Layer 3108. In step 6012, the nearest NE with Transport Layer3108 (UDP) and sufficient buffer space is identified. In step 6014, theNE receives all clear (no fault) information from Data Link Layer 3104.Finally, in step 6016, the NE resumes the transmission rate using UDP.

The system and methods of the illustrative embodiments allow bandwidthallocation, resource management, and troubleshooting across Ethernet orcommunications domains. Network performance information about the linestate of the Ethernet network may be used in conjunction with or toadjust Connection Admission Control (CAC) policies and devices inreal-time such that bandwidth and services across Ethernet domains arecontrolled. The network performance information may also be used toisolate nodes that are failing or sources of trouble in order to makenetwork corrections. The changes, adjustments, fixes, work-arounds thatmay be implemented are available across communications domains elementswith different operators and equipment in a carrier grade Ethernetnetwork. Various access nodes, such as Broadband Remote Access Servers(BRAS), Broadband Digital Loop Carriers (BBDLC), Cable Modem TerminationSystems (CMTS), or routers and switches, may use the illustrativemethods to manage services and/or bandwidths by tracking both the sharednetwork-side trunk state and the individual subscriber-side line statesas a state repository for the network systems.

FIG. 61 is an example of an Ethernet network 6100 in accordance with anillustrative embodiment of the present invention. FIG. 61 illustratesthe Ethernet network 6100 with a number of access nodes 6101, 6102,6104, and 6106 in communication with Connection Admission Control (CAC)engines 6108, 6110, 6112, and 6114. Each of these CAC engines 6108,6110, 6112, and 6114, devices, or elements is connected to a data stream6116 that communicates between and through the access nodes 6101, 6102,6104, and 6106 using packet streams 6118, 6120, 6122, and 6124.

Ethernet is a network protocol and local area network (LAN) technologyused for sending and receiving data packets across the Ethernet network6100. CAC engines 6108, 6110, 6112, and 6114 control and adjust theconnection bandwidth in order to accommodate the necessary communicationstream. CAC engines 6108, 6110, 6112, and 6114 may be hardware and/orsoftware elements or process performed thereby to take actions duringconnection initiation or re-allocation for strategically controllingcongestion. Frequently, the CAC engines 6108, 6110, 6112, and 6114 maybe used to determine whether or not to allow a new connection, throttlebandwidth, or to load balance across the Ethernet network 6100. The CACengines 6108, 6110, 6112, and 6114 may communicate messages, alerts,alarms, commands, data, and other information with one another. In oneembodiment the CAC engines may contain transmission path state real-timebandwidth and other PM information. The CAC policy engine may include athreshold trigger based upon the PM information or the CAC engine maydynamically change over-subscription rules for the bandwidth reservationportion of the CAC engine. These new states and state triggers areequivalent to CAC engine states that may be polled or otherwiseinteracted with by EMS systems and other protocols. In anotherembodiment, a connection may be accepted only if sufficient resourcesare available to establish the connection end-to-end with its requiredquality of service. For example, in one embodiment, for a new connectionto be accepted, the contractual quality of service of existingconnections and customers served by the network may not be adverselyaffected by the new connection.

In some cases, CAC engine 6108 may be used to control CAC engine 6114.Provisionally, the CAC engines may be applicable to a port or atransmission path. Each CAC engine 6108, 6110, 6112, and 6114 mayspecify permissions and authorizations for how and when it may beaccessed. The permissions may include authentications, passwords, andidentifications so that each CAC engine 6108, 6110, 6112, and 6114 doesnot have unlimited access to each of the other CAC engines 6108, 6110,6112, and 6114. For example, if CAC engine 6114 receives a bandwidththrottling request from CAC engine 6108, the CAC engine 6114 may ensurethat the CAC engine 6108 is part of an authentication list of devices,nodes, EVCs, and elements with permission to adjust bandwidth for CACengine 6114.

The data stream 6116 may include video, data, voice, or other multimediapacket streams. Each of the packet streams 6118, 6120, 6122 and 6124 mayrepresent separate Ethernet virtual connections that send and receivepackets through the data stream 6116. The CAC engines may control thepackets being placed on the data stream 6116 or being taken off the datastream 6116 and sent to a user or customer.

FIG. 62 is an example of an Ethernet network 6200 in accordance with anillustrative embodiment of the present invention. Ethernet network 6200may be a particular implementation of Ethernet network 6100 of FIG. 61.Ethernet network 6200 includes Ethernet domains 6202 and 6204,maintenance endpoints 6206, 6208, 6210, and 6212 and maintenanceintermediate endpoints 6214, 6216, 6218, 6220, and 6222.

The Ethernet domains 6202 and 6204 represent Ethernet networkscontrolled by separate operators that have maintenance end-points 6206,6208, 6210, and 6212 as defined by the IEEE 802.1AG standards. Themaintenance end-points 6206 and 6208 are in the first Ethernet domain6202 and the maintenance end-points 6210 and 6212 are in the Ethernetdomain 6204. In an illustrative embodiment of the present invention,information that traverses the entire network between maintenanceend-points 6206 and 6212 may be available at every end-point andintermediate point that is connected between maintenance end-points 6206and 6212. For instance, maintenance intermediate end-point 6216 may haveinformation from itself as well as maintenance intermediate end-points6214, 6218, 6220, and 6222 and maintenance end-points 6206, 6208, 6210,and 6212. The information may describe the total packet rate orreal-time data packet rate, average packet rates, packet rates for thestreams from access node users and any other statistics related to thecommunications capability and health of the network. The networkperformance information may be contained in the Ethernet layer 2real-time packet flows.

FIG. 63 is an example of a CAC engine configuration in accordance withan illustrative embodiment of the present invention. FIG. 63 includesdata stream 6302, access node 6304, CAC engine 6306, network performanceinformation 6308, user packet stream 6310, customer 6311, line stateinformation 6312, and correlation engine 6314. The network performanceinformation 6308 may be updated from the line state information 6312available on a network node, store in a table on the network orextracted from one or more packets in the data stream 6302, such as aPIP data packet. The network performance information 6308 may includenetwork statistics, including performance, such as real-time bandwidth,of packets including real-time content. For example, the networkperformance information 6308 may specify statistics calculated from theline state information 6312 including a provisioned rate, real-timepacket rate, and average packet rate, real-time and total bandwidthusage. The line state information 6312 may represent the data andinformation obtained from operation measurements as described by thecurrent invention.

The user packet stream 6310 is controlled by the CAC engine 6306allowing data to be sent and received between the customer 6311 and thedata stream 6302 as determined by the network performance information6308 and the policies established by the operator of the communicationsnetwork via the CAC engine policy modification based on the Performanceinformation 6308.

In one embodiment, the access node 6304 may include a network cache thatstores information, such as movies, songs, games, and other data, thatmay be accessed by the CAC engine 6306 and delivered to the data stream6302 for immediate use by other network users. The network cache mayalso store network performance information for historical use andsubsequent reference. For example, if a node within the network hasrepeatedly had problems, the historical data may be used to link theproblem to certain events, parameters, or factors. The CAC engine 6306may determine the adequacy of the data stream 6302 to be able toaccommodate the current movie or other data packets requested by a useron the network. In addition the CAC engine 6306 may determine if thedata information requested was legitimately requested and authorized asset by previously determined network policies.

FIG. 64 is an example of PIP packet flow of network performanceinformation in accordance with an illustrative embodiment of the presentinvention. Packet flow 6400 of FIG. 64 shows the network performanceinformation obtained using the teachings of the present invention.Packet flow 6400 illustrates data and PIP packet flow across accessnodes 6402, 6404, and 6406, network performance information 6408, 6410,and 6412. The PIP packet flow 6400 includes information about thenetwork line state at intermediate points and at end-points in thenetwork. The PIP packet flow may be a particular implementation of datastream 6302 of FIG. 63.

As the network performance information 6408 is passed through eachaccess node 6402, 6404, and 6406 in the packet flow 6400, additionalinformation is added by each access node 6402, 6404, and 6406. It isunderstood that each network segment and path may have a PIP performanceand utilization PM flow and measure. For example, as shown, initialnetwork performance information 6408 may be a single PIP packet. Later,as the network performance information reaches the access node 6404,additional network performance information 6410 may be added. Thenetwork performance information 6410 may be combined into a single PIPpacket or multiple PIP packets may be used. At access node 6406, the PIPpacket includes network performance information 6408, 6410, and 6412.Information may be added for any number of access nodes despite thelimited examples shown.

As a result, the PIP packets are passed from and to or through each ofthe access nodes, devices, and other elements of a network communicationsystem. Thus, at any given access node the network performanceinformation for each prior node may be easily ascertained and analyzedas needed. Similar information on the performance of the network mayalso be available in like manner from a PIP packet flow 6400 going theother direction because PIP packets may flow in both directions in thenetwork. Alternatively, network performance information may be obtainedfrom or utilized in a central database, EMS server, NOC, CCM or othercentral resource in communication with a CAC engine or access node.

FIG. 65 is an example of stored network performance informationassociated with access nodes in accordance with an illustrativeembodiment of the present invention. FIG. 65 details the networkperformance numbers and statistics that may be available or stored ateach access node 6502, such as access nodes 6402, 6404, and 6406 of FIG.64. In another embodiment, the network performance information table maybe a compilation of data stored in a central network device, generalstate engine, or other element or component that is accessible bydifferent nodes and processors within the network. The networkperformance information table may also be stored in the correlationengine 6314 of FIG. 63. The centralized table may alternatively beupdated when the network experiences problems, other tables have dataoverflow, or processing elements are unable to process the networkperformance information fast enough. The correlation engine may sendalerts or alarms to access nodes or a network control center to corrector troubleshoot network issues.

The network performance information which may include numbers andstatistics may be part of a table or matrix that describes the networkin terms of packet count, packet delay, packet loss for the total,real-time, and/or non-real-time flows, total real-time, andnon-real-time bandwidth, effective packet rate, jitter, latency,out-of-order packets, quality of service, carrier identification, orother parameters 6500 that describes the important characteristics ofthe end-to-end network. The different numbers, values, and measurementsmay be used to calculate in the overall quality of the network byviewing these numbers singularly at each access node or collectively asa network packet loss statistic or other parameter. The networkperformance information may be collected over time to provide an averagetime-bound network value.

In another instance, numbers, such as the provisioned and or availablepacket bandwidth rate (pn) and the real-time bandwidth packet rate inuse (mn), may be used to provide a measure to individual CAC enginesthat may indicate that access to the network should be accepted ordeclined. For instance, if the network performance value: (pn−mn) ispositive, it may indicate that there is capacity on the network for theaccess node packet stream number n. If the value (pn−mn) is negative orzero, it may indicate that the CAC engine declines the packet streamfrom access node n.

In another instance, individual network values such as the real-timebandwidth packet rate (kn) may be averaged over the network to provide areal-time average network packet rate such as: ((k1+k2+kn)/n). Thereal-time average may be used to provide a real-time use statistic to anoverall monitoring center that may automatically, or through operatorassistance, admit or reject additional packet streams through the CACengines. For example, the real-time average may be used to rejectadditional packet streams requesting to join a certain program eventthat is oversubscribed at the server. As a result, additional users maybe rejected. It should be understood that packet rate can be bandwidthin use, packet counts, or a combination of both.

In another embodiment, a remote server, such as a “video on demand”server, may be dedicated to providing video files for any number ofcustomers. In response to a CAC engine receiving network performanceinformation, via, for example, PIP packets, for node segments associatedwith the remote server indicative of the real-time bandwidth and otherperformance information, such as packet loss or congestion, are used toobtain a utilization performance measure in relation to the assumedusage and oversubscription rates or other performance issues associatedwith receiving content from such remote server. The CAC engine maythrottle the allocation of bandwidth to devices requesting contentthereby modifying the bandwidth that may be accessible by each customer,IP address, or other element. Such CAC engine may be located at an IPservice or access broadband node gateway point used by the remote serveror at access points used by CPEs to access content from such remoteserver over a packet network. For example, CAC engines with theappropriate permissions may be used to throttle the bandwidth of theremote server or to specify priorities. For example, a CAC engine in aremote location may be used to specify that a first CAC engine or aspecific network device of high priority may have unlimited bandwidthaccess to the remote server, but all other CAC engines, CPEs, orspecific network devices may only have a designated percentage ofavailable bandwidth.

FIG. 66 is a flowchart of a process for allocating network resources inaccordance with an illustrative embodiment of the present invention. Theprocess of FIG. 66 may be implemented by an CAC engine.

The process begins by gathering network performance informationregarding line and trunk transmission performance and utilization states(step 6602) and/or other network performance information. The networkperformance information may include performance numbers, data,utilization information, or statistical information calculated therefromregarding real-time and non-real-time data passing through the datastream of the communications network. The network performanceinformation may be gathered using a PIP packets and PM collection pointsthat has been updated as it reaches each access node within a networksuch as an Ethernet network. The network performance and utilizationinformation gathered in step 6602 may also be stored for subsequentanalysis.

Next, the CAC engine controls the network resources (step 6604) inresponse to received network performance and utilization information.The network resources are controlled based on the performanceinformation particularly for dynamic resource allocation, diagnosis, andtroubleshooting. For example, if a node, device, link access point, orother node segment is encountering problems, the VOD session controllerredirects the CAC engine as it may reroute the IP Service point trafficaround the node experiencing problems via addressing, or server nameresponse. The CAC engine may also perform load balancing betweendifferent CAC engines. For example, a single customer may be connectedto the communications network through different CAC engines. Based ontraffic between the customer through the different streams, loadbalancing may be performed so that bandwidth is more efficientlyutilized across the CAC engines. The same types of balancing is commonlyperformed using current protocols and applications, such as Bit Torrent.These conventional protocols were designed to acquire portions ofcontent, such as a movie from multiple sources, concurrently. Theseprotocols by-pass the rate limiting effect imposed by egress rateshaping functions at VOD or content servers. These protocols can havesignificant affect on the performance of the aggregate path. PIP packetscan detect the impacts of the use of these types of parallel protocolsand be used to invoke any of the traffic management functionalitydescribed in accordance with the principles of the present invention.

For purposes of this example, load balancing may refer to a throttlingof bandwidth by two or more VOD servers responding to CAC engines and/orthe routing of traffic or sessions by two or more VOD servers respondingto CAC engines transmission path utilization and performance stateinformation to even out traffic that is directed at two or more accesspoints to a network Thus, a customer network may access a larger packetnetwork, such as the Internet, through connections associated withnetwork access points. The bandwidth of data passing through eachnetwork access point may be controlled by one or more CAC enginesworking in concert basing load balancing on the transmission stateinformation for the paths under the governance of reach CAC engine. Tobalance the amount of traffic through each of the network access points,one or more CAC engines may cause traffic intended for one of suchnetwork access points that is approaching full load or a overloadedstate to be rerouted or redirected to another network access point thatis not experiencing as much traffic. The one or more network accesspoints may be access points to the same network, such as the Internet,or may alternatively access different packet networks. However, even ifsuch network access points are egresses from a customer network into twodifferent packet networks, both of such packet networks may eventuallyallow a connection to an IP address or other network address locatedoutside of the customer network. For example, one of the network accesspoints may allow egress into a first network that contains the IPaddress to which a data communication is addressed, while the othernetwork access point may allow egress to a second network that is thenconnected to a third network, that is, in turn, connected to the firstnetwork including the IP address to which a communication is addressed.In such a manner, even if network access points and associated CACmanagers are not connected to the same external network both suchnetwork access points may allow egress of a data packet in a manner thatallows the data packet to eventually be communicated to the target IPaddress.

Although the foregoing is described generally with respect to reroutingof traffic and the distribution of new sessions based on actualperformance and utilization information of the transmission paths thesystem intelligently distributes the bandwidth across two or morenetwork access points by one or more CAC managers, it should beunderstood that more complicated schemes of load balancing can beutilized that involve algorithms associated with bandwidth reservationand allocation, the throttling of bandwidth, the rerouting of traffic toalternative network access points, known connection paths to an IPaddress located outside of the network through one or more networkaccess points, or any combination of the foregoing. In yet anotherembodiment, load balancing may include a determination of particularapplication data included in packets intended to be communicated throughnetwork access points such that such load balancing may be accomplishednot only with respect to traffic generally, but with respect to trafficassociated with a particular application or class of applications. Forexample, the most readily apparent example of the need for such specificload balancing may be with regard to the load balancing of real-timesession packets associated with applications that perform real-time ornear real-time content communications. The handling of real-time packetsmay cause more network performance issues than the handling ofnon-real-time packets as a result of the need to minimize latency andjitter associated with such real-time packets. In this case, the goal ofthe CAC function is to balance the load of real-time traffic acrossmultiple network paths to optimize network performance. It should beunderstood that other real-time flows may exist in the egress trunk ofwhich the CAC engine may not have knowledge. In order to balance thereal-time traffic, the performance and utilization state information maybe present in or accessible to the CAC engine. Thus, complex loadbalancing schemes that take into account real-time data packets andnon-real-time data packets in balancing the two categories of datapackets across one or more network access points by CAC managers thattake into account network performance information regarding externalnetworks and the connection paths available therein may greatly enhancethe user experience associated with applications using real-time contentand offer a unique way to respond to performance issues identified fromnetwork performance information in order to address such issues andenhance the general performance of the network.

In one embodiment, a CAC engine may manage one or more additional CACengines within the network or in a secondary network in response toreceived network performance information. For example, a CAC engine maycontrol bandwidth usage in an interconnected network. For example, thenetwork performance information or instructions to a CAC engine locatedin another network may be carried or “piggybacked” from a first networkto a second network, via, for example, PIP packets for allowing asecondary CAC engine to control a CAC engine in the first network.

In one embodiment, the CAC engine may throttle sessions or restrict theamount of allocated bandwidth (for a smaller codec) based on the amountof bandwidth available for a customer to access a network (step 6606) inresponse to received network performance information. Bandwidth requestsmay be granted, throttled, and bandwidth reserved and allocated based onexternal or internal factors that are affecting the communicationsnetwork. For example, bandwidth may be throttled based on interferencethrough a CAC engine that relies on a wireless transmission point. Thebandwidth through the CAC engine may be throttled to accommodate theavailable connection speeds and limiting factors of the wirelesstransmission point.

A CAC engine reserves and allocates bandwidth for customers at a networkaccess point. For example, the customer may have a service levelagreement or quality of service requirement specifying certainparameters and resources for which the customer is paying. For example,the customer may have reserved 10 megabits/second for real-timestreaming video. If the bandwidth of real-time data packets dedicatedfor the customer is running at 12 megabits/second, the CAC engine maythrottle or adjust the customer stream so that only 10 megabits/secondis provided to the customer. As a result, bandwidth may become availableto other customers that are paying more for increased bandwidth or aguarantee that bandwidth will be available from the communicationservice provider at any time. Some service level agreements and qualityof service provisions allow for refunds or discounts if available ratesor bandwidth levels drop below specified thresholds as provided to thecustomer by the communication network service provider.

In another embodiment, the additional 2 megabits/second of bandwidth maybe allocated on a “best efforts” or similar non-guaranteed basis. As aresult, if bandwidth is available, the customer may be provided theentire 12 megabits/second, otherwise the customer is provided only the10 megabits/second that are guaranteed the customer. In anotherembodiment, the customer may be provided the entire 12 megabits/secondand is charged a premium rate for the data overage. The customer mayalso be billed for the amount of time that the bandwidth used exceeds 10megabits/second. The updates may be sent from the CAC engine to abilling database. The customer service level agreement may specifydifferent rates, charges, guarantees, quality of service, or servicelevel agreements for both real-time and non-real-time content.

The network performance and utilization information located in the CACengine may also be used to enforce usage limitations, or track customerbandwidth to identify an IP address that is monopolizing or overusingresources. Once the customer or IP address found to be a over-usingresources has been located via threshold mechanisms set on the lineperformance and utilization state information for that line, by the CACengine or other network process or device, the CAC engine may throttlesessions, shut down or alter shaping windows for that customer flow, orsend a message to session controllers or customer GUI interfaces toprovide a usage warning message, or otherwise limit the customer'saccess to the network in order to preserve bandwidth and bandwidthavailability across the network or meet business objectives. Forexample, a student in a dorm that is streaming too much real-time datafor a movie may have their real-time bandwidth limited to providebandwidth for other students or customers.

In another example, the bandwidth percentages or rates available to acustomer may be increased in step 6606 in response to received networkperformance information. The bandwidth available for customers acrossthe Ethernet network may be dynamically adjusted based on service levelagreements, guarantees, performance representations, type of packets,performance indicators, and other parameters or factors. The bandwidthmay be adjusted for access nodes, customers, devices, softwareapplications, and IP addresses. In many cases, the throttling of step6606 is performed based on the type of data, including real-time andnon-real-time data. For example, a customer may have desired rates andpercentages of dedicated bandwidth for real-time and non real-time datapackets. In many cases, real-time Voice over Internet Protocol (VoIP)may be considered a higher priority than regular Internet traffic. Step6606 may also shift bandwidth and network traffic for load balancingbetween access nodes for better network performance.

Throttling requests and other configuration and maintenance changesimplemented by a CAC engine may be implemented by inserting commands orother data in the PIP packet. By inserting the changes to be made in thePIP packet, the CAC engine may perform the changes in-band and does notneed access to an out-of-band communication connection in order toimplement the desired allocation and reservation. Alternatively,allocation and reservation changes made by a CAC engine may occurout-of-band using an alternative communication line or medium.

The PIP packets may also include data for load balancing between CACengines, access nodes, and other communications elements. The networkperformance information, control signal communication, and/or PIPpackets may also be sent to access nodes, CAC engines, a networkoperation center using enhanced messaging service (EMS) or othermessaging protocols. The PIP packets may also specify real-timethresholds, percentages, and parameters that may be used to regulate thecommunications network. The PIP packets may add reservation andallocation information that regulates the control process by customer,identification, IP address, or program application. For example, a PIPpacket may specify that a CAC engine is to dedicate five percent ofavailable bandwidth to real-time data from IP address 128.063.254.

FIG. 67 is a flowchart of a process for correcting failure of networkresources in accordance with an illustrative embodiment of the presentinvention. The process of FIG. 67 may be implemented by an access node.

The process begins in step 6702 by gathering network performanceinformation regarding line state. Step 6702 may be performed aspreviously described in step 6602 of FIG. 66. Next, the access nodecompares thresholds against the network performance information in step6704. The comparison of step 6704 may be performed by a correlationengine that is part of the access node or independent from the accessnode. The network performance information may be compared against atable, matrix numbers, or statistics. The results of the comparison maybe compared against rule-based statistics. The results may be used todetermine the status and performance of the communications networkincluding software and hardware components within the network.

The access node determines whether there is an access node experiencingfailure in step 6706. The determination may be made based on thecomparison of the network performance information to thresholds in step6704. If there is not an access node experiencing failure, the processterminates. If there is an access node experiencing failure in step6706, the access nodes corrects the failure for the access node in step6708 with the process terminating thereafter. In one example, if aproblem or failure is detected in step 6706, the problem may becorrected manually or automatically by a network control center. Forexample, if there is a failure, the access point may send a correctionmessage, alert, or alarm to the network control center so that thefailing or problem node may be fixed with a software patch, reboot,maintenance order, replacement, or work-around.

The network corrections may occur in any number of ways based on theparameters, designated preferences, rules, and policies. The correctionsmay be temporary or permanent fixes. The solution to the failure may notalways be easily fixed. In one embodiment, the failure may be correctedby sending a notification to other networks that are dependent on thenetwork. In another embodiment, the traffic may be rerouted throughdifferent CAC engines, access nodes, or networks in order to correctnetwork issues.

In another embodiment, the network correction may involve rerouting datathrough different CAC engines, networks, and access nodes. The data maybe rerouted to preserve quality of service and performance of thenetwork. The CAC engine may also request or generate trace informationfor the failing access node. The CAC engine may also request that thefailing access node or surrounding access nodes or device provideadditional information. The additional information may supplement thedata provided by the PIP packet to diagnose and remedy the problem. TheCAC engine may also ping the network access nodes to determine how dataand information is flowing within the network and whether each accessnode is available. For example, a number of access nodes may besystematically pinged by the CAC engine to determine which nodes arestill responsive.

The CAC engine may broadcast a message out to other access nodes, a NOC,EMS system, CCM, or other network device or process that identifies thefailing access node and the associated problem. The message mayspecifically indicate the failing node, network performance informationfor the failing node, and a network solution, remedy, or work-around, ifapplicable. The message may be sent specifically to a network operationcenter, performance log or table, or a rule based engine that may usenetwork topology to select or provide solutions for the problem. Therule-based engine may also be used to select the next step taken addressthe problem, such as send a text message to a network administrator.

The network performance information may be archived or stored. Thehistorical data may be used to reserve and allocate bandwidth changes.The historical data may be used to implement different control andalgorithm changes based on an analysis of the historical data. Forexample, the historical data may reflect particular network performanceissues that occur at specific times of day or in response to specificevents. Such specific network performance issues can then be addressedby changing the throttling, reservation, and allocation scheme ofparticular CAC managers associated with particular network access pointsduring such times or prior to such events. Thus, a CAC manager mayrespond to reservation requests associated with a particular networkaccess point during particular times of the day with the allocation ofbandwidth to requesting IP addresses that is less than the allocationthat is made at other times of day. For example, a CAC engine mayrespond to bandwidth reservation requests by allocating only 50% of thebandwidth requested in such an allocation request.

Such historical data may also be used to implement particular loadbalancing algorithms between different network access points using oneor more CAC managers. For example, if a particular network access pointreceives a number of requests for bandwidth allocation associated withreal-time data packets for an application using real-time content, suchas a video conferencing application, allocations associated with suchreal-time applications may be directed alternatively to a differentnetwork access point based on the particular load balancing algorithm.Thus, historical data related to network performance information can bestored by CAC manager and utilized to change bandwidth throttling,reservation, and allocation algorithms that take into account the levelof granularity of network performance information obtained by such CACmanager.

In one embodiment, the CAC manager may completely block reservationrequests received that are associated with particular IP addresses,applications, or network protocols. In yet another embodiment, a CACmanager may receive network performance information associated withenhanced levels of jitter experienced in the external network, and inresponse thereto, may limit requests to reserve bandwidth that areassociated with a SIP protocol or itself request that a particularnetwork device located at a particular IP address utilize a lower ratecodec than may have previously requested before allocating bandwidth inresponse to such bandwidth reservation requests.

As previously discussed, one CAC engine may communicate with one or moreCAC engines located elsewhere within a customer network or even withinan external network. The historical data may, therefore, also be used torequest or command that alternative throttling, reservation, andallocation algorithms or instructions be utilized by other CAC managers.For example, corrections for different nodes, devices, or elements ofthe communications network may be made to avoid disrupting as fewcustomers and network traffic as possible.

In one embodiment, the CAC engine may access a table of networkperformance information that is stored within the engine or that isstored remotely within the network or within another communicationsnetwork. The CAC engine may access data within the table to identify theproblem for troubleshooting purposes.

In one embodiment, network performance information that is obtainedusing PIP packets, from any of the tables disclosed herein, or any othermanner, may be utilized to change the way in which future networkperformance information is gathered, collected, or analyzed. Forexample, if an analysis of network performance information reveals aproblem in a particular network or portion of a network then thefrequency at which PIP packets are sent through that portion of thenetwork may be increased. Alternatively, the level of networkperformance information collected by such PIP packets may be increasedsuch that more data and more types of data are collected by PIP packetsas they are routed through that network or portion of the network. Inyet another embodiment, the normal routing of PIP packets through anetwork may be changed to avoid a severed communication link obtain moreinformation about a particular portion of the network encounteringproblems, or to send instructions to such portion of the network tocause such portion or the notes therein to collect additionalinformation, run diagnostic routines, otherwise troubleshoot theproblem, or implement changes in the ways that network devices withinsuch portion of the network are configured or operate. For example, aparticular network node such as a switch may be instructed to beginparticular congestion control behavior, reroute traffic, or any otherappropriate solution or work-around to a particular problem identifiedby network performance information.

In one embodiment, a network device may be instructed by informationwithin a PIP packet to reboot itself, refresh a routing table, increasethe amount of data buffered within such network device, or otherwisebegin, change, or terminate any process or operation within such networkdevice. In one embodiment, the control regimen for managing PIP packetflow may be changed entirely in response to a change in networkperformance information. For example, a network or portion of thenetwork over which PIP packets were being sent every five seconds may beincreased in frequency to every one second or tenth of a second tocollect more information about a particular network performance problem.

Alternatively, in an embodiment that does not include the use of PIPpackets, each network node may be instructed to update its own internaltable or update a table of network performance information used as anessential resource by one or more networks on a more frequent basis orwith increasing levels of detail of network performance information. Inanother embodiment, when PIP packets or other updates regarding networkperformance information were previously generated only in response toevents or triggers or requests received from another network device,such network node may instead be instructed to generate such PIP packetsor other updates on a routine basis until a problem has been resolved.All the foregoing may be implemented at any time in response to atrigger, or may alternatively be scheduled at a particular time based onan instruction. Similarly, the entire PIP packet system and/or system ofupdates regarding network performance information may revert back to anormal mode of operation after a network performance problem has beenresolved, if a period of time has elapsed, or it receives additionalinstructions from a central network resource or other network noteddevice.

As a result of the foregoing or as the result of other events notdescribed above, the payload of PIP packets may be altered. For example,the PIP packet may be instructed to obtain network performanceinformation at a node that was previously a pass-through node from whichnetwork performance information was not obtained. In another embodiment,PIP packets may include a payload of specific instructions to a networkdevice at a particular network node to change the operation of suchnetwork node. In yet another embodiment, a PIP packet that wasoriginally instructed only to obtain network performance informationwith regard to latency, packet loss, and jitter, may instead beinstructed to obtain a full description of all network performanceinformation that is available, alternative network performanceinformation, or some level of network performance information betweenfull network performance information and the limited amount of networkperformance information it would normally receive.

The use of injecting instructions into a PIP packet allows the use ofPIP packets as much more than a simple reporting system of networkperformance information. Instead, it allows an inband system for sendinginstructions, initiating processes, or otherwise configuring theparameters or operation of a network. Although not expressly describedherein, the foregoing schemes to modify PIP packets or other methods ofreporting network performance information may be directed through anetwork outside of the packet network (such as LAN) generating the PIPpackets or otherwise seeking to obtain network performance information.For example, if a particular network is experiencing delays, packetloss, or jitter in a large number of data packets that originate from aparticular outside network the network can increase the number of PIPpackets that are sent to such outside network in order to obtain networkperformance information from such network or may request that suchoutside network generate more of its own PIP packets directed to suchnetwork from such outside network. In such a manner, the network mayascertain specific performance problems in the outside network, and ifnecessary, reroute calls or other data communications through othernetworks instead of such outside network.

In one embodiment, network performance information may be utilized by aprocessor to determine the best gateway, access point,network-to-network interface, or other network egress point for aparticular data packet or session of data packets. For example, if aparticular network has five different egress points to an outsidenetwork, such as a PSTN, such five egress points represent fivedifferent ways in which a VoIP call or other communication may be routedin order to access such PSTN or other outside network. Within thenetwork itself, there may be five network connection paths between acustomer access point making a VoIP call connection request and each ofsuch egress points. Thus, there may be twenty-five different routes inwhich a VoIP call may be connected from the customer access pointsthrough the network in order to communicate with the PSTN or otheroutside network. An EMS system, inter-network system, CCM, router, orother network device may utilize network performance information todetermine which of such twenty-five potential connections paths willoffer the best quality of service for the customer attempting to connectthe VoIP. Such network device may also consider which of the twenty-fiveconnection paths least negatively affects the performance of theremainder of such network.

Such decision may be made in response to known information about networkperformance outside of the network within the PSTN. For example, if oneof the egress points is known to be in a portion of the PSTNexperiencing significant performance issues, the five potentialconnections associated with such point of egress may be avoided.Likewise, if ten out of the twenty-five potential connection paths gothrough a common core switch within the network that is experiencingproblems, those ten options can likewise be discarded.

Each of the twenty-five potential network connection paths between acustomer's access point and the PSTN may be rated, graded, or otherwisecompared to determine the best possible connection path for eithercustomer quality service and/or general network performance. Forexample, the present application discloses different ways to rateparticular node segments of a network by assigning a particular ratingto them. Different ratings can be assigned to each node segment based ondifferent criteria such as latency, jitter, packet loss, percentage ofreal-time traffic, real-time bandwidth, or any other parameter. Thus,each node segment may have different grades or ratings in differentareas.

Using the one or more grades or ratings for each of the node segments,many different algorithms may be used to calculate the best overall pathbetween the customer's access point to the network and the PSTN or otheroutside network. For example, ratings associated with jitter could begiven one weighting factor and averaged across all node segments locatedbetween the customer's network access point and the PSTN, which may, inturn, be added to another weighted factor with respect to latency thatrepresents the average latency grade across each of the node segmentsbetween the customer's network access point and the PSTN. In anotherembodiment, connection paths between the network access point and thePSTN may be examined to determine the highest or lowest rating that hasbeen given to any node segment located in such path which such highestor lowest rating being then assigned to the entire path.

In some cases, actual measurements may be utilized to determine whetheror not to use a path. For example, the average jitter experienced by areal-time application packet passing through a particular node segmentmay be added to such averaged jitter of each of the other node segmentslocated along such connection path to given overall average jitter thatmay be experienced by a real-time application communicating packetsalong such path. Similarly, the overall latency of a path can becalculated. The overall rating of the twenty-five possible connectionpaths may be compared to each other either in aggregate or based ondifferent factors and otherwise weighted, filtered, or analyzed, todetermine the best possible connection path for the customer's qualityof service and/or the general performance of the network. Somealgorithms may weight the effect on overall network performances beingmore important than the customer's quality of service as long as, forexample, a particular minimum quality of service is reached. Algorithmsmay also take into account the guarantees or service level agreementsthat the network provider may have with a particular customer such thatthe network provider may meet its commitments to the customer and doesnot have to pay to the customer any service level credits, penalties,damages, or expenses.

In one embodiment of the present invention, network performanceinformation associated with a particular network that has been collectedusing PIP packets, retrievals from tables of network performanceinformation, or any other means of obtaining network performanceinformation may be utilized by a particular application being executedby a client of such network. In such an embodiment, information from atable or a PIP packet may be communicated into the particularapplication using the application's protocol, an injection from anotherOSI layer into the application layer, an XML interface, or any othersuitable means. Such application may then have instructions that executein response to receiving particular network performance information.

For example, an application may be a network computer game wherein thegamer is looking for the best possible connection with the least amountof latency and packet loss that is available between the gamer's clientand a remote server hosting the on-line game. The application may obtainnetwork performance information in order to itself calculate such bestconnection or instead query a central network resource, such as a VPCEdescribed earlier, to respond with the best network connection possible.In one embodiment, an application may display network performanceinformation to a user of a client. In another embodiment, an applicationmay cause an instruction to be injected into a PIP packet or similarpacket to obtain specific network performance information from a remotenetwork or node segment or to cause changes in a remote network or nodesegment to enhance the users experience of a particular application.

In one embodiment, an application may affirmatively monitor networkperformance information in order to determine a better networkconnection than the connection it is currently using and switch to suchbetter connection as soon as it is detected. Alternatively, a PIP packetor other instruction may be sent to a particular application in responseto network performance issues believed to be caused by such applicationin order to reduce the amount of bandwidth utilized by such application,change the operation of such application, terminate such application,restart such application, or otherwise change the application to reduceany negative effects on network performance and/or potentially purchaseduser experience.

In one embodiment, the network performance information and differentlevels of detail thereof may be utilized with one or more of the variouspermission schemes described herein or any other suitable security-basedaccess to network performance information may be used to share networkperformance information between two or more networks. In one embodiment,the information, or a subset thereof, from all networks participating ina global network such as the Internet may be updated to a centralresource and shared among all participants in the global network. Inanother embodiment, networks may share their network performanceinformation with any network to which they have a network-to-networkinterface. In another embodiment, a network may combine other networksnetwork performance information with its own network performanceinformation in a table or other central or distributed resource.

Although the permission's table described herein relative to FIG. 17B isillustrated and described to show access to different levels of networkperformance information, a similar table may be utilized to establishwhat access, testing, instructions, commands, and other communicationsan outside network is permitted to make within a particular networkoperator's network. For example, a particular network operator may allowan outside network to manage or send certain instructions to CACmanagers, or certain CAC managers, located within the network operator'snetwork. In another embodiment, a network operator may permit an outsidenetwork to control certain aspects of the operation of a device of suchnetwork operators that forms part of a network-to-network interface withsuch outside network. In yet another embodiment, the network operatormay allow an outside network to control a particular trunk, nodesegment, or connection path used solely or primarily to route traffic toor from such outside network.

In yet another embodiment, particular emergency circumstances may changethe level of access that a network operator gives to outside networksover its own network devices and node segments. For example, aparticular emergency event may trigger a command to remove all outsidenetwork access to a network operator's network, whether with respect toreceiving network performance information, issuing commands andresponses thereto, executing troubleshooting or testing routines, or anyother communication inconsistent with normal data communication andnetwork operation. In response to another event, a network operator mayconfigure permissions to allow an outside network to obtain enhanced oradditional network performance information from within a networkoperator's network.

Any combination of the foregoing may be utilized to allow interconnectednetworks and the operators thereof to have increased visibility into theoperation of each of the interconnected networks and offer some measureof allowing each network to control certain aspects of networkperformance to address problems determined by analyzing such receivednetwork performance information.

In one embodiment, the network performance information is collected,shared, and analyzed automatically by network devices and processes andinstructions, commands, further troubleshooting, work-arounds, or othersolutions may be automatically generated using suitable ruled-basedengines or algorithms to reroute traffic, throttle or block traffic,change the configuration of network devices, or implement any otherchange in one or more networks to resolve problems associated with theinterconnection of data packets across such networks. Different schemes,automated processes, rules, and algorithms may be utilized with respectto real-time data packet communication as opposed to non real-time datapacket communication or overall data packet communication.

Although network performance information is described as being stored intables and carried by PIP packets and otherwise monitored with respectto raw network performance information such as actual measurements ofpacket loss, latency, bandwidth, real-time bandwidth, jitter, or anyother measured or detected item of data associated with networkperformance, such data may also be stored in tables or communicated viaPIP packets after being converted, analyzed, summarized, averaged, orotherwise used in a calculation or statistical analysis to form deriveddata. Such derived data shall also be considered network performanceinformation.

In one embodiment, statistical information may be kept regarding networkperformance information that is gathered over a certain interval oftime. During such certain interval of time, many measurements of thesame data may be made. Different data can be collected and determined atthe end of such interval. For example, each measurement itself may bestored, an average of a measurement, such as bandwidth, may be stored, amode associated with the most frequent bandwidth range or value measuredmay be stored such as for example, the most frequently measured range ofbandwidth.

In another embodiment, the peak bandwidth or other network performancedata may be determined and stored. In yet another embodiment,information may be presented as network performance information thatestablishes the percentage of such interval that the bandwidth waswithin a particular range. For example, during a measurement interval ofone second, 100 measurements may be taken in response, for example, toreceiving 100 PIP packets at a particular network node. Such 100measurements may be analyzed and: (i) an average packet loss, latencyand jitter determined, (ii) a peak latency, packet loss, and jitterdetermined, (iii) a mode or mode range of latency, packet loss, andjitter may be determined, or (iv) any other statistical measure of anydata collected as a result of the receipt of such PIP packet may bedetermined.

For the same 100 sets of network performance data generated, percentagesand ranges may be utilized to enhance visibility of the networkperformance information. For example, if 50 of the 100 measurementstaken for bandwidth of real-time data packets are determined to bebetween zero and 100 Mbps, 20 measurements are determined to be betweena range of 100 Mbps and 500 Mbps, and the remaining 20 measurements aredetermined to be greater than 500 Mbps. Network performance informationmay be generated to indicate that during one second interval of time,less than 100 Mbps of real-time data packets will be received 50% of thetime between 100 and 500 Mbps of real-time data packets will be received30% of the time, and greater than 500 Mbps of real-time data packetswill be received 20% of the time. As described above, filtering may alsobe utilized. For example, the five lowest measured and the five highestmeasured amounts of bandwidth may be discarded before doingcalculations. In another example, measurements may be tracked as afunction of time. In yet another embodiment, the rate of change ofbandwidth, latency, packet loss, jitter, or any other measurement may beutilized to generate indications associated with increasing or decliningcongestion, increasing or declining latency, increasing or decliningjitter, or increasing or declining packet loss from one interval to thenext interval. Thus, for example, network performance information may begenerated that shows the degree of change in any data measured by thenode between intervals that may be useful to identify trends in networkperformance or otherwise detect declining network performance ordeclining traffic through a particular network element that may beindicative of a problem elsewhere in the network.

In one embodiment, an identifier associated with a graphical element maybe included as network performance information such that when networkperformance information is received by a network operation's center,Internet work system, EMS system, or any other network resource ornetwork device, the graphical element may be displayed as an up arrow,down arrow, blockage symbol, impaired symbol, normal symbol, or anyother symbol indicative of network performance information. Thus, forexample, an operator of such a resource may see displayed an up arrowindicating that congestion at a particular network switch is increasing,which may alert an operator to look more closely at the remainingnetwork performance information associated with one or more networkdevices.

The previous detailed description is of a small number of embodimentsfor implementing the invention and is not intended to be limiting inscope. The following claims set forth a number of the embodiments of theinvention disclosed with greater particularity.

The invention claimed is:
 1. A network communications device,comprising: an input/output (I/O) unit configured to communicate datapackets to and from a communications network; and at least one moduleconfigured to: count total data packets being communicated via said I/Ounit during a time period to determine a total data packet count;distinguish the total data packets between real-time data packets thatinclude real-time content and non-real-time data packets that do notinclude real-time content; count at least one of data packets thatinclude real-time content to determine a real-time data packet count anddata packets that do not include real-time content to determine anon-real-time data packet count that are communicated via said I/O unitduring the time period, wherein the real-time data packet count is acount of only data packets that include real-time content, wherein thetotal data packet count equals the real-time data packet count plus thenon-real-time data packet count; determine bandwidth of total datapackets being communicated via said I/O unit; determine bandwidthcorresponding to the real-time data packet count; determine bandwidthreservation use information; concatenate an indicia representative ofthe bandwidth of data packets corresponding to the real-time data packetcount and the bandwidth reservation use information to produce amodified data packet; and transmitting the modified data packet to asecond network communication device.
 2. The network communicationsdevice according to claim 1, wherein the data packet to which theindicia is concatenated is a performance information data packet.
 3. Thenetwork communications device according to claim 1, wherein thedetermination of the bandwidth of data packets including real-timecontent is based on a number of performance information data packetsreceived during the time period.
 4. The network communications deviceaccording to claim 3, wherein the performance information data packetsare IEEE standard 802.1AG data packets.
 5. The network communicationsdevice according to claim 1, wherein said at least one module includes amodified ITU-T Y.1731 module configured to separately count the totaldata packets and data packets including real-time content.
 6. Thenetwork communications device according to claim 1, wherein the thirdnetwork communications device is configured to monitor performance ofother network nodes.
 7. The network communications device according toclaim 1, wherein said at least one module is further configured toconvert the data packets from a first communications protocol to asecond communications protocol.
 8. The network communications deviceaccording to claim 1, wherein said at least one module is furtherconfigured to operate as a firewall.
 9. The network communicationsdevice according to claim 1, wherein said at least one module is furtherconfigured to determine at least one transmission performancecharacteristic of the data packets including real-time content beingcommunicated via said I/O unit.
 10. The network communications deviceaccording to claim 1, wherein communication of the indicia is eventtriggered.
 11. A method for monitoring transmission characteristics on apacket network, said method comprising: monitoring data packets beingcommunicated over a network node segment of a packet network;distinguish the data packets into real-time data packets that includereal-time content and non-real-time data packets that do not includereal-time content; counting the real-time data packets beingcommunicated over the network node segment during a time period;determining bandwidth of the real-time data packets during the timeperiod; counting total data packets being communicated over the networknode segment during the time period; determining bandwidth of total datapackets being communicated over the network node segment during the timeperiod; storing an indicia representative of bandwidth of real-time datapackets and total data packets; determining at least one transmissionperformance characteristic of the real-time data packets; concatenatethe indicia representative of the at least one transmission performancecharacteristic of the communicated real-time data packets to a datapacket payload to produce a modified data packet; and transmitting themodified data.
 12. The method according to claim 11, wherein determiningthe at least one transmission performance characteristic of real-timedata packets includes determining at least one of percent loss, jitter,and delay.
 13. The method according to claim 11, further comprising:determining a time period from performance information data packetsbeing communicated over the network node segment; and determining thebandwidth of the real-time and total data packets being communicatedover the network node segment based on the time period determined fromthe counted performance information data packets.
 14. The methodaccording to claim 13, further wherein counting performance informationdata packets includes counting 802.1 AG data packets.
 15. The methodaccording to claim 13, further comprising communicating an indicia ofthe bandwidth of the data packets including real-time content and totaldata packets to a network controller.
 16. The method according to claim11, wherein concatenating the indicia into a data packet includingconcatenating the indicia into a performance information data packet.17. The method according to claim 11, wherein communicating indicia ofthe at least one transmission performance characteristic includescommunicating the indicia in response to an event occurring.
 18. Themethod according to claim 11, wherein communicating the indicia of theat least one transmission performance characteristic in response to anevent occurring includes determining that a threshold has been crossedby the at least one transmission performance characteristic.
 19. Anetwork communications device, comprising: an input/output (I/O) unitconfigured to communicate data packets over a network node segment; amemory unit configured to store network performance informationassociated with real-time data packets and total data packetscommunicated via said I/O unit; and a processor configured to executeinstructions to: distinguish the total data packets between thereal-time data packets that include real-time content and non-real-timedata packets that do not include real-time content, count the real-timedata packets being communicated over the network node segment during atime period; determine bandwidth of the real-time data packets duringthe time period; count total data packets being communicated over thenetwork node segment during the time period; determine bandwidth oftotal data packets being communicated over the network node segmentduring the time period; communicate an indicia representative ofbandwidth of the real-time data packets and total data packets; andconcatenate the network performance information associated withreal-time data packets and total data packets to a data packet payloadto produce a modified data packet.
 20. The network communications deviceaccording to claim 19, wherein the processor is further configured toexecute instructions to collect the network performance informationassociated with real-time data packets communicated via said I/O unit.21. The network communications device according to claim 19, wherein theprocessor is further configured to execute instructions to: countperformance information data packets being communicated over the networknode segment; and determine the bandwidth of the real-time and totaldata packets being communicated over the network node segment based onthe counted performance information data packets.
 22. The networkcommunications device according to claim 21, wherein the performanceinformation data packets are IEEE 802.1AG data packets.
 23. The networkcommunications device according to claim 19, wherein the networkperformance information includes at least one of packet loss, jitter,and delay associated with data packets including real-time contentcommunicated via said I/O unit communications.
 24. The networkcommunications device according to claim 19, wherein the memory unit isfurther configured to store information indicative of bandwidth of thereal-time data packets and total data packets communicated during a timeperiod.